Methods and Systems for Generating and Using Localization Reference Data

ABSTRACT

Methods and systems for improved positioning accuracy relative to a digital map are disclosed, and which are preferably used for highly and fully automated driving applications, and which may use localisation reference data associated with a digital map. The invention further extends to methods and systems for the generation of localisation reference data associated with a digital map.

FIELD OF THE INVENTION

This invention relates, in certain aspects and embodiments, to methodsand systems for improved positioning accuracy relative to a digital map,and which is needed for highly and fully automated driving applications.Such methods and systems may use localisation reference data associatedwith a digital map. In further aspects, the present invention relates tothe generation of localisation reference data associated with a digitalmap, including the format of the reference data, and the use of thereference data. For example, embodiments of the invention relate to theuse of the reference data through a comparison to sensed data from avehicle to accurately position the vehicle on the digital map. Otherembodiments relate to the use of the reference data for other purposes,not necessarily in techniques which also use sensed data from a vehicle.For example, further embodiments relate to the use of the generatedreference data for reconstructing a view from a camera associated with avehicle.

BACKGROUND OF THE INVENTION

It has become common in recent years for vehicles to be equipped withnavigation devices, either in the form of portable navigation devices(PNDs) that can be removably positioned within the vehicle or systemsthat are integrated into the vehicle. These navigation devices comprisea means for determining the current position of the device; typically aglobal navigation satellite system (GNSS) receiver, such as GPS orGLONASS. It will be appreciated, however, that other means may be used,such as using the mobile telecommunications network, surface beacons orthe like.

Navigation devices also have access to a digital map representative of anavigable network on which the vehicle is travelling. The digital map(or mathematical graph, as it is sometimes known), in its simplest form,is effectively a database containing data representative of nodes, mostcommonly representative of road intersections, and lines between thosenodes representing the roads between those intersections. In moredetailed digital maps, lines may be divided into segments defined by astart node and end node. These nodes may be “real” in that theyrepresent a road intersection at which a minimum of 3 lines or segmentsintersect, or they may be “artificial” in that they are provided asanchors for segments not being defined at one or both ends by a realnode to provide, among other things, shape information for a particularstretch of road or a means of identifying the position along a road atwhich some characteristic of that road changes, e.g. a speed limit. Inpractically all modern digital maps, nodes and segments are furtherdefined by various attributes which are again represented by data in thedatabase. For example, each node will typically have geographicalcoordinates to define its real-world position, e.g. latitude andlongitude. Nodes will also typically have manoeuvre data associatedtherewith, which indicate whether it is possible, at an intersection, tomove from one road to another; while the segments will also haveassociated attributes such as the maximum speed permitted, the lanesize, number of lanes, whether there is a divider in-between, etc. Forthe purposes of this application, a digital map of this form will bereferred to as a “standard map”.

Navigation devices are arranged to be able to use the current positionof the device, together with the standard map, to perform a number oftasks, such as guidance with respect to a determined route, and theprovision of traffic and travel information relative to the currentposition or predicted future position based on a determined route.

It has been recognised, however, that the data contained within standardmaps is insufficient for various next generation applications, such ashighly automated driving in which the vehicle is able to automaticallycontrol, for example, acceleration, braking and steering without inputfrom the driver, and even fully automated “self-driving” vehicles. Forsuch applications, a more precise digital map is needed. This moredetailed digital map typically comprises a three-dimensional vectormodel in which each lane of a road is represented separately, togetherwith connectivity data to other lanes. For the purposes of thisapplication, a digital map of this form will be referred to as a“planning map” or “high definition (HD) map”.

A representation of a portion of a planning map is shown in FIG. 1,wherein each line represents the centreline of a lane. FIG. 2 showsanother exemplary portion of a planning map, but this time overlaid onan image of the road network. The data within these maps is typicallyaccurate to within a metre, or even less, and can be collected usingvarious techniques.

One exemplary technique for collecting the data to build such planningmaps is to use mobile mapping systems; an example of which is depictedin FIG. 3. The mobile mapping system 2 comprises a survey vehicle 4, adigital camera 40 and a laser scanner 6 mounted on the roof 8 of thevehicle 4. The survey vehicle 2 further comprises a processor 10, amemory 12 and a transceiver 14. In addition, the survey vehicle 2comprises an absolute positioning device 2, such as a GNSS receiver, anda relative positioning device 22 including an inertial measurement unit(IMU) and a distance measurement instrument (DMI). The absolutepositioning device 20 provides geographical coordinates of the vehicle,and the relative positioning device 22 serves to enhance the accuracy ofthe coordinates measured by the absolute positioning device 20 (and toreplace the absolute positioning device in those instances when signalsfrom the navigation satellites cannot be received). The laser scanner 6,the camera 40, the memory 12, the transceiver 14, the absolutepositioning device 20 and the relative positioning device 22 are allconfigured for communication with the processor 10 (as indicated bylines 24). The laser scanner 6 is configured to scan a laser beam in 3Dacross the environment and to create a point cloud representative of theenvironment; each point indicating the position of a surface of anobject from which the laser beam is reflected. The laser scanner 6 isalso configured as a time-of-flight laser range-finder so as to measurea distance to each position of incidence of the laser beam on the objectsurface.

In use, as shown in FIG. 4, the survey vehicle 4 travels along a road 30comprising a surface 32 having road markings 34 painted thereon. Theprocessor 10 determines the position and the orientation of the vehicle4 at any instant of time from position and orientation data measuredusing the absolute positioning device 20 and the relative positioningdevice 22, and stores the data in the memory 12 with suitabletimestamps. In addition, the camera 40 repeatedly captures images of theroad surface 32 to provide a plurality of road surface images; theprocessor 10 adding a timestamp to each image and storing the images inthe memory 12. The laser scanner 6 also repeatedly scans the surface 32to provide at least a plurality of measured distance values; theprocessor adding a timestamp to each distance value and stores them inthe memory 12. Examples of the data obtained from the laser scanner 6are shown in FIGS. 5 and 6. FIG. 5 shows a 3D view, and FIG. 6 shows aside view projection; the colour in each picture being representative ofthe distance to the road. All the data obtained from these mobilemapping vehicles can be analysed and used to create planning maps of theportions of the navigable (or road) network travelled by the vehicles.

It has been recognised by the Applicant that in order to use suchplanning maps for highly and fully automated driving applications, it isnecessary to know the position of a vehicle relative to the planning mapto a high degree of accuracy. The traditional technique of determiningthe current location of a device using navigation satellites orterrestrial beacons provides an absolute position of the device to anaccuracy of around 5-10 metres; this absolute position is then matchedto a corresponding position on the digital map. While this level ofaccuracy is sufficient for most traditional applications, it is notsufficiently accurate for next generation applications, where positionsrelative to the digital map are required at sub-metre accuracy even whentravelling at high speeds on the road network. An improved positioningmethod is therefore required.

The Applicant has also recognised that there is a need for improvedmethods of generating localisation reference data associated with adigital map e.g. for providing a “planning map”, which may be used indetermining the position of a vehicle relative to the map, as well as inother contexts.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by a referenceline associated with the navigable element, each pixel of the at leastone depth map being associated with a position in the reference planeassociated with the navigable element, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along a predetermined direction from the associated positionof the pixel in the reference plane; and

associating the generated localisation reference data with the digitalmap data.

It will be appreciated that the digital map (in this, and any otheraspects or embodiments of the invention) comprises data representativeof navigable elements of a navigable network, e.g. roads of a roadnetwork.

In accordance with the first aspect of the invention, localisationreference data is generated associated with one or more navigableelements of a navigable network represented by a digital map. Such datamay be generated in respect of at least some, and preferably all of thenavigable elements represented by the map. The generated data provides acompressed representation of the environment around the navigableelement. This is achieved using at least one depth map, which isindicative of the environment around the element projected on to areference plane defined by a reference line, which in turn, is definedwith respect to the navigable element. Each pixel of the depth map isassociated with a position in the reference plane, and includes a depthchannel representing the distance to the surface of an object in theenvironment along a predetermined direction from the position of thepixel in the reference plane.

Various features of the at least one depth map of the localisationreference data will now be described. It will be appreciated that suchfeatures may alternatively or additionally be applied to the at leastone depth map of the real time scan data that is used in certain furtheraspects or embodiments of the invention, to the extent that they are notmutually exclusive.

The reference line associated with the navigable element, and which isused to define the reference plane may be set in any manner with respectto the navigable element. The reference line is defined by a point orpoints associated with the navigable element. The reference line mayhave a predetermined orientation with respect to the navigable element.In preferred embodiments the reference line is parallel to the navigableelement. This may be appropriate for providing localisation referencedata (and/or real time scan data) relating to the lateral environment ona side or sides of the navigable element. The reference line may belinear or non-linear i.e. depending whether the navigable element isstraight or not. The reference line may include straight and non-linear,e.g. curved portions, e.g. to remain parallel to the navigable element.It will be appreciated that in some further embodiments, the referenceline may not be parallel to the navigable element. For example, asdescribed below, the reference line may be defined by a radius centredon a point associated with a navigable element e.g. a point on thenavigable element. The reference line may be circular. This may thenprovide a 360 degree representation of an environment around a junction.

The reference line is preferably a longitudinal reference line, and maybe, for example, an edge or boundary of the navigable element or a lanethereof, or a centre line of the navigable element. The localisationreference data (and/or real time scan data) will then provide arepresentation of the environment on a side or sides of the element. Thereference line may lie on the element.

In embodiments, the reference line may be linear even when the navigableelement is curved, since a reference line of the navigable element, suchas the edge or centreline of the navigable element, and the associateddepth information, may undergo a mapping to a linear reference line.Such a mapping or transformation is described in more detail in WO2009/045096 A1; the entire content of which is incorporated herein byreference.

The reference plane defined by the reference line is preferablyorientated perpendicular to a surface of the navigable element. Areference plane as used herein, refers to a 2 dimensional surface, whichmay be curved or non-curved.

Where the reference line is a longitudinal reference line parallel tothe navigable element, the depth channel of each pixel preferablyrepresents the lateral distance to a surface of an object in theenvironment.

Each depth map may be in the form of a raster image. It will beappreciated that each depth map represents the distance along apredetermined direction from surfaces of objects in the environment tothe reference plane for a plurality of longitudinal positions andelevations i.e. corresponding to the position of each pixel associatedwith the reference plane. The depth map comprises a plurality of pixels.Each pixel of the depth map is associated with a particular longitudinalposition and elevation in the depth map, e.g. raster image.

In some preferred embodiments the reference plane is defined by alongitudinal reference line parallel to the navigable element, and thereference plane is orientated perpendicularly to a surface of thenavigable element. Each pixel then includes a depth channel representingthe lateral distance to a surface of an object in the environment.

In preferred embodiments, the at least one depth map may have a fixedlongitudinal resolution and a variable vertical and/or depth resolution.

In accordance with a second aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by alongitudinal reference line parallel to the navigable element andorientated perpendicularly to a surface of the navigable element, eachpixel of the at least one depth map being associated with a position inthe reference plane associated with the navigable element, and the pixelincluding a depth channel representing the lateral distance to a surfaceof an object in the environment along a predetermined direction from theassociated position of the pixel in the reference plane, preferablywherein said at least one depth map has a fixed longitudinal resolutionand a variable vertical and/or depth resolution; and

associating the generated localisation reference data with the digitalmap data.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

Regardless of the orientation of the reference line, reference plane andthe line along which the environment is projected onto the referenceplane, in accordance with the invention in its various aspects andembodiments, it is advantageous for the at least one depth map to have afixed longitudinal resolution and a variable vertical and/or depthresolution. The at least one depth map of the localisation referencedata (and/or real time scan data) preferably has a fixed longitudinalresolution and a variable vertical and/or depth resolution. The variablevertical and/or depth resolution is preferably non-linear. The portionsof the depth map, e.g. raster image, closer to the ground and closer tothe navigable element (and hence closer to a vehicle) may be shown in ahigher resolution than portions of the depth map, e.g. raster image,higher above the ground and further away from the navigable element (andhence vehicle). This maximises the information density at heights anddepths which are more important for detection by vehicle sensors.

Regardless of the orientation of the reference line and plane, or theresolution of the depth map along the various directions, the projectionof the environment on to the reference plane is along a predetermineddirection, which may be selected as desired. In some embodiments theprojection is an orthogonal projection. In these embodiments the depthchannel of each pixel represents the distance to a surface of an objectin the environment from the associated position of the pixel in thereference plane along a direction normal to the reference plane. Thus,in some embodiments in which the distance represented by the depthchannel is a lateral distance, the lateral distance is along a directionnormal to the reference plane (although non-orthogonal projections arenot confined to cases in which the depth channel relates to a lateraldistance). The use of an orthogonal projection may be advantageous insome contexts, as this will have the result that any height informationis independent of the distance from the reference line (and hencereference plane).

In other embodiments, it has been found that it may be advantageous touse a non-orthogonal projection. Thus, in some embodiments of theinvention in any of its aspects, unless mutually exclusive, (whether ornot the predetermined distance is a lateral distance) the depth channelof each pixel represents the distance to a surface of an object in theenvironment from the associated position of the pixel in the referenceplane along a direction that is not normal to the reference plane. Theuse of a non-orthogonal projection has the advantage that informationregarding surfaces oriented perpendicular to the navigable element maybe preserved (i.e. where the reference line is parallel to the element).This may be achieved without needing to provide additional data channelsassociated with pixels. Information regarding objects in the vicinity ofthe navigable element may therefore be captured more effectively, and ingreater detail, but without needing to increase storage capacity. Thepredetermined direction may be along any desired direction relative tothe reference plane, such as at 45 degrees.

The use of a non-orthogonal projection has also been found to be usefulin preserving a greater amount of information about surfaces of objectsthat may be detected by a camera or cameras of a vehicle underconditions of darkness, and is therefore particularly useful incombination with some aspects and embodiments of the invention in whicha reference image or point cloud is compared to an image or point cloudobtained based upon real time data sensed by camera(s) of a vehicle.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by a referenceline parallel to the navigable element, each pixel of the at least onedepth map being associated with a position in the reference planeassociated with the navigable element, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along a predetermined direction from the associated positionof the pixel in the reference plane, wherein the predetermined directionis not normal to the reference plane; and

associating the generated localisation reference data with digital mapdata indicative of the navigable element.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with the invention in any of its aspects or embodiments,the localisation reference data (and/or real-time scan data) is based onscan data obtained by scanning the environment around the navigableelement using one or more sensors. The one or more scanners may compriseone or more of; a laser scanner, a radar scanner and a camera, e.g. asingle camera or a pair of stereo cameras.

Preferably the distance to the surface of an object represented by thedepth channel of each pixel of the localisation reference data (and/orthe real-time scan data) is determined based upon a group of a pluralityof sensed data points, each indicative of a distance to the surface ofan object along the predetermined direction from the position of thepixel. The data points may be obtained when performing a scan of theenvironment around the navigable element. The group of sensed datapoints may be obtained from one or more types of sensor. However, insome preferred embodiments the sensed data points comprise or consist ofa group of data points sensed by a laser scanner or scanners. In otherwords, the sensed data points comprise or consist of laser measurements.

It has been found that using an average of multiple sensed data pointsin determining the distance value for the depth channel for a givenpixel may lead to erroneous results. This is because there is alikelihood that at least some of the sensed data points indicative ofthe position of the surface of an object from the reference plane alongthe applicable predetermined direction, and which are considered to mapto a particular pixel, may relate to surfaces of different objects. Itwill be appreciated that, due to the compressed data format, an extendedarea of the environment may map to the area of the pixel in thereference plane. A considerable amount of sensed data, i.e. number ofsensed data points may therefore be applicable to that pixel. Withinthat area, there may be objects located at different depths relative tothe reference plane, including objects which may overlap another objectover only a short distance in either dimension, e.g. trees, lampposts,walls, as well as moving objects. The depth values to the surface of anobject represented by the sensor data points applicable to a particularpixel may therefore exhibit considerable variance.

In accordance with the invention in any of its aspects or embodiments,in which the distance to the surface of an object represented by thedepth channel of each pixel of the localisation reference data (and/orthe real-time scan data) is determined based upon a group of a pluralityof sensed data points, each indicative of a sensed distance to thesurface of an object along the predetermined direction from the positionof the pixel, preferably the distance represented by the depth channelof a pixel is not an average value based on the group of a plurality ofsensed data points. In preferred embodiments the distance represented bythe depth channel of a pixel is a closest sensed distance to the surfaceof an object from among the group of sensed data points, or a closestmode value obtained using a distribution of the sensed depth values. Itwill be appreciated that the closest value or values detected are likelyto most accurately reflect the depth of the surface of an object to thepixel. For example, consider the case in which a tree is located betweena building and a road. Different sensed depth values applicable to aparticular pixel may be based on detection of either the building or thetree. If all of these sensed values were taken into account to providean average depth value, the average value would indicate that the depthto the surface of an object measured from the pixel was somewherebetween the depth to the tree and the depth to the building. This wouldlead to a misleading depth value for the pixel, which would lead toproblems when correlating real-time vehicle sensed data to the referencedata, and could potentially be dangerous, as it is of criticalimportance to know with confidence how close objects are to the road. Incontrast, the closest depth value or closest mode values are likely torelate to the tree, rather than the building, reflecting the trueposition of the closest object.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by a referenceline associated with the navigable element, each pixel of the at leastone depth map being associated with a position in the reference planeassociated with the navigable element, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along a predetermined direction from the associated positionof the pixel in the reference plane, wherein the distance to the surfaceof an object represented by the depth channel of each pixel isdetermined based upon a group of a plurality of sensed data points, eachindicative of a sensed distance to the surface of an object along thepredetermined direction from the position of the pixel, and wherein thedistance to the surface of the object represented by the depth channelof the pixel is the closest distance, or closest mode distance, based onthe group of sensed data points; and

associating the generated localisation reference data with the digitalmap data.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with the invention in any of its aspects or embodiments,each pixel (in the localisation reference data and/or the real time scandata) includes a depth channel representing the distance to a surface ofan object in the environment. In preferred embodiments each pixelincludes one or more further channels. This may provide the depth mapwith one or more additional layers of information. Each channel ispreferably indicative of a value of a property obtained based upon oneor more sensed data points, and preferably a group of a plurality ofsensed data points. The sensed data may be obtained from one or moresensors as earlier described. In preferred embodiments the or each pixelincludes at least one channel indicative of a value of a given type ofsensed reflectivity. Each pixel may comprise one or more of: a channelindicative of a value of a sensed laser reflectivity; and a channelindicative of a value of a sensed radar reflectivity. The sensedreflectivity value of the pixel indicated by a channel relates to thesensed reflectivity in the applicable portion of the environmentrepresented by the pixel. The sensed reflectivity value of the pixel ispreferably indicative of the sensed reflectivity at around a distancefrom the reference plane corresponding to the depth of the pixel fromthe reference plane indicated by the depth channel of the pixel, i.e.the sensed reflectivity at around the depth value for the pixel. Thismay then be taken to be indicative of the relevant reflectivity propertyof the object present at that depth. Preferably the sensed reflectivityis a mean reflectivity. The sensed reflectivity data may be based upon areflectivity associated with the same data points used to determine thedepth value, or of a larger set of data points. For example, areflectivity associated with sensed depth values applicable to thepixel, and other than those closest values which are preferably used indetermining the depth value for the depth channel, may be additionallytaken into account.

In this way, a multi-channel depth map, e.g. raster image, is provided.Such a format may enable a larger amount of data relating to theenvironment surrounding the navigable element to be more efficientlycompressed, facilitating storage and processing, and providing theability to carry out improved correlation with real-time data sensed bya vehicle under different conditions, and without the vehiclenecessarily needing to have the same type of sensors as used ingenerating the reference localisation data. As will be described in moredetail below, such data may also help in reconstructing data sensed by avehicle, or an image of the environment around the navigable elementthat would be obtained using a camera of the vehicle, under particularconditions, e.g. at night. For example, radar or laser reflectivity mayenable those objects that would be visible under particular conditions,e.g. at night, to be identified.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by a referenceline associated with the navigable element, each pixel of the at leastone depth map being associated with a position in the reference planeassociated with the navigable element, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along a predetermined direction from the associated positionof the pixel in the reference plane, wherein each pixel further includesone or more of: a channel indicative of a value of a sensed laserreflectivity; and a channel indicative of a value of a sensed radarreflectivity; and

associating the generated localisation reference data with the digitalmap data.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with the invention in any of its aspects or embodiments,other channels associated with pixels may alternatively or additionallybe used. For example, further channels may be indicative of one or moreof: a thickness of the object at around the distance along thepredetermined direction from the reference plane from the position ofthe pixel indicated by the depth channel of the pixel; a density ofreflected data points at around the distance along the predetermineddirection from the reference plane from the position of the pixelindicated by the depth channel of the pixel; a colour at around thedistance along the predetermined direction from the reference plane fromthe position of the pixel indicated by the depth channel of the pixel;and a texture at around the distance along the predetermined directionfrom the reference plane from the position of the pixel indicated by thedepth channel of the pixel. Each channel may comprise a value indicativeof the relevant property. The value is based upon applicable sensor dataobtained, which may be obtained from one or more different types ofsensor as appropriate, e.g. a camera for colour or texture data. Eachvalue may be based upon multiple sensed data points, and may be anaverage of values from said multiple sensed data points.

It will be appreciated that while the depth channel is indicative of thedistance of an object from the reference plane at the position of apixel along a predetermined direction, the other channels may beindicative of other properties of the object, e.g. the reflectivity ofthe object, or its colour, texture, etc. This may be useful inreconstructing scan data that can be expected to have been sensed by avehicle and/or camera images taken by a vehicle. Data indicative of thethickness of an object may be used to recover information relating tosurfaces of the object perpendicular to the navigable element, where anorthogonal projection of the environment on to the reference plane isused. This may provide an alternative to the embodiments described abovefor determining information relating to such surfaces of objects, whichuse a non-orthogonal projection.

In many embodiments, the localisation reference data is used to providea compressed representation of the environment to a side or sides of anavigable element, i.e. to provide a side depth map. The reference linemay then be parallel to the navigable element, with the depth channel ofa pixel indicating a lateral distance of the object surface from thereference plane. However, the use of a depth map may also be helpful inother contexts. The Applicant has realised that it would be useful toprovide a circular depth map in the region of a junction, e.g.crossroads. This may provide an improved ability to position a vehiclewith respect to the junction, e.g. cross roads, or, if desired, toreconstruct data indicative of the environment around the junction, e.g.cross roads. A 360 degree representation of the environment around thejunction is preferably provided, although it will be appreciated thatthe depth map need not extend around a full circle, and may thereforeextend around less than 360 degrees. In some embodiments the referenceplane is defined by a reference line defined by a radius centred on areference point associated with a navigable element. In theseembodiments the reference line is curved, and preferably circular. Thereference point is preferably located on the navigable segment at ajunction. For example, the reference point may be located at a centre ofthe junction, e.g. crossroads. The radius defining the reference linemay be chosen as desired, e.g. depending upon the size of the junction.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map representing elements of a navigable network, thelocalisation reference data providing a compressed representation of anenvironment around at least one junction of the navigable networkrepresented by the digital map, the method comprising, for at least onejunction represented by the digital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the junction projected on to areference plane, said reference plane being defined by a reference linedefined by a radius centred on a reference point associated with thejunction, each pixel of the at least one depth map being associated witha position in the reference plane associated with the junction, and thepixel including a depth channel representing the distance to a surfaceof an object in the environment along a predetermined direction from theassociated position of the pixel in the reference plane; and

associating the generated localisation reference data with digital mapdata indicative of the junction.

As described in relation to the earlier embodiments, the junction may bea crossroads. The reference point may be located at a centre of thejunction. The reference point may be associated with a node of thedigital map representing the junction, or a navigable element at thejunction. These further aspects or embodiments of the invention may beutilised in combination with side depth maps representing theenvironment to the side of navigable elements away from the junction.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with the invention in any of its aspects or embodimentsrelating to the generation of localisation reference data, the methodmay comprise associating the generated localisation reference data inrespect of a navigable element or junction with the digital map dataindicative of the element or junction. The method may comprise storingthe generated localisation data associated with the digital map data,e.g. in association with the navigable element or junction to which itrelates.

In some embodiments, the localisation reference data may comprise arepresentation, e.g. reference scan of the lateral environment on a leftside of the navigable element and a right side of the navigable element.The localisation reference data for each side of the navigable elementmay be stored in a combined data set. Thus, the data from multiple partsof the navigable network may be stored together in an efficient dataformat. The data stored in the combined data set may be compressed,allowing data for more parts of the navigable network to be storedwithin the same storage capacity. Data compression will also allow areduced network bandwidth to be used should the reference scan data betransmitted to the vehicle over a wireless network connection. However,it will be appreciated that the localisation reference data does notnecessarily need to relate to the lateral environment on either side ofthe navigable element. For example, as discussed in certain embodimentsabove, the reference data may relate to the environment surrounding ajunction.

The present invention also extends to a data product storing thelocalisation reference data generated in accordance with any of theaspects or embodiments of the invention.

The data product in any of these further aspects or embodiments of theinvention may be of any suitable form. In some embodiments the dataproduct may be stored on a computer readable medium. The computerreadable medium may be, for example, a diskette, CD ROM, ROM, RAM, flashmemory or hard disk. The present invention extends to a computerreadable medium comprising the data product in accordance with theinvention of any of its aspects or embodiments.

The localisation reference data generated in accordance with theinvention in any of the aspects or embodiments relating to thegeneration of such data may be used in various manners. In the furtheraspects relating to the use of the data, the step of obtaining thereference data may extend to generating the data, or typically compriseretrieving the data. The reference data is preferably generated by aserver. The steps of using the data are preferably performed by a devicesuch as a navigation device or similar, which may be associated with avehicle.

In some preferred embodiments the data is used in determining a positionof a vehicle relative to the digital map. The digital map thus comprisesdata representative of navigable elements along which the vehicle istravelling. The method may comprise obtaining the localisation referencedata associated with the digital map for a deemed current position ofthe vehicle along a navigable element of the navigable network;determining real time scan data by scanning the environment around thevehicle using at least one sensor, wherein the real time scan datacomprises at least one depth map indicative of an environment around thevehicle, each pixel of the at least one depth map being associated witha position in the reference plane associated with the navigable element,and the pixel including a depth channel representing the distance to asurface of an object in the environment along the predetermineddirection from the associated position of the pixel in the referenceplane as determined using the at least one sensor; calculating acorrelation between the localisation reference data and the real timescan data to determine an alignment offset between the depth maps; andusing the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map. It will be appreciated that the localisation reference datathat is obtained relates to the navigable element along which thevehicle is travelling. The depth map of the localisation reference data,which is indicative of the environment around the navigable element ishence indicative of the environment around the vehicle.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of navigable elements ofa navigable network along which the vehicle is travelling, the methodcomprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, said reference plane being defined bya reference line associated with the navigable element, each pixel ofthe at least one depth map being associated with a position in thereference plane associated with the navigable element along which thevehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, wherein the real time scan datacomprises at least one depth map indicative of an environment around thevehicle, each pixel of the at least one depth map being associated witha position in the reference plane associated with the navigable elementalong which the vehicle is travelling, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along the predetermined direction from the associatedposition of the pixel in the reference plane as determined using the atleast one sensor;

calculating a correlation between the localisation reference data andthe real time scan data to determine an alignment offset between thedepth maps; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In the further aspects and embodiments of the invention relating to theuse of localisation reference data and real time scan data indetermining the position of a vehicle, the current position of thevehicle may be a longitudinal position. The real-time scan data mayrelate to a lateral environment around the vehicle. The depth map forthe localisation reference data and/or the real time sensor data willthen be defined by a reference line parallel to the navigable element,and including a depth channel representing the lateral distance to thesurface of an object in the environment. The determined offset may thenbe a longitudinal offset.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of a junction throughwhich the vehicle is travelling, the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle in the navigable network,wherein the location reference data comprises at least one depth mapindicative of an environment around the vehicle projected on to areference plane, said reference plane being defined by a reference linedefined by a radius centred on a reference point associated with thejunction, each pixel of the at least one depth map being associated witha position in the reference plane associated with the junction throughwhich the vehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, wherein the real time scan datacomprises at least one depth map indicative of an environment around thevehicle, each pixel of the at least one depth map being associated witha position in the reference plane associated with the junction, and thepixel including a depth channel representing the distance to a surfaceof an object in the environment along the predetermined direction fromthe associated position of the pixel in the reference plane asdetermined using the at least one sensor;

calculating a correlation between the localisation reference data andthe real time scan data to determine an alignment offset between thedepth maps; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with another aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of navigable elements ofa navigable network along which the vehicle is travelling, the methodcomprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof a navigable network, wherein the location reference data comprises atleast one depth map indicative of an environment around the vehicle,each pixel of the at least one depth map being associated with aposition in a reference plane associated with the navigable element,said reference plane being defined by a longitudinal reference lineparallel to the navigable element and orientated perpendicularly to asurface of the navigable element, and each pixel including a depthchannel representing the lateral distance to a surface of an object inthe environment, optionally wherein said at least one depth map has afixed longitudinal resolution and a variable vertical and/or depthresolution;

obtaining sensor data by scanning the environment around the vehicleusing at least one sensor;

determining real time scan data using the sensor data, wherein the realtime scan data comprises at least one depth map indicative of anenvironment around the vehicle, each pixel of the at least one depth mapbeing associated with a position in the reference plane associated withthe navigable element, and each pixel including a depth channelrepresenting the lateral distance to a surface of an object in theenvironment as determined from the sensor data, optionally wherein saidat least one depth map has a fixed longitudinal resolution and avariable vertical and/or depth resolution;

calculating a correlation between the localisation reference data andthe real time scan data to determine an alignment offset between thedepth maps; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In the further aspects of the invention relating to the use of thelocalised reference data, the data may be generated in accordance withany of the earlier aspects of the invention. The real time scan dataused in determining the position of the vehicle, or otherwise, should beof a corresponding form to the localisation reference data. Thus, thedepth map determined will comprise pixels having positions in areference plane defined with respect to a reference line associated witha navigable element in the same manner as the localised reference data,to enable the real time scan data and localised reference data to becorrelated with one another. The depth channel data of the depth map maybe determined in a corresponding manner to that of the reference data,e.g. without using an average of the sensed data, and thus may comprisea closest distance to a surface from a plurality of sensed data points.The real-time scan data may include any additional channels. Where thedepth map of the localisation reference data has a fixed longitudinalresolution and a variable vertical and/or depth resolution, the depthmap of the real time scan data may also have such resolution.

Thus, in accordance with these aspects or embodiments of the invention,there is provided a method of continually determining a position of avehicle relative to a digital map; the digital map comprising datarepresentative of navigable elements (e.g. roads) of a navigable network(e.g. road network) along which the vehicle is travelling. The methodcomprises receiving real time scan data obtained by scanning anenvironment around the vehicle; retrieving localisation reference dataassociated with the digital map for a deemed current position of thevehicle in relation to the digital map, (e.g. wherein the localisationreference data comprises a reference scan of the environment around thedeemed current position), optionally wherein said reference scan hasbeen obtained throughout the digital map from at least one device whichhas previously travelled along the route; comparing the real time scandata to the localisation reference data to determine an offset betweenthe real time scan data and the localisation reference data; andadjusting the deemed current position based on said offset. The positionof the vehicle relative to the digital map can therefore always be knownto a high degree of accuracy. Examples in the prior art have attemptedto determine the position of a vehicle by comparing collected data withknown reference data for pre-determined landmarks along a route.However, the landmarks may be sparsely distributed on many routes,resulting in significant errors in the estimation of the vehicle'sposition when it is travelling between the landmarks. This is a problemin situations such as highly automated driving systems, where sucherrors can cause catastrophic consequences such as vehicle crashincidents resulting in serious injury or loss of life. The presentinvention, in certain aspects at least, solves this problem by havingreference scan data throughout the digital map and by scanning theenvironment around the vehicle in real time. In this way, the presentinvention may allow real time scan and reference data to be comparedsuch that the position of the vehicle relative to the digital map isalways known to a high degree of accuracy.

In accordance with a further aspect of the invention there is provided amethod of determining a longitudinal position of a vehicle relative to adigital map, the digital map comprising data representative of navigableelements of a navigable network along which the vehicle is travelling,the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof a navigable network, wherein the location reference data comprises anoutline of objects in an environment around the vehicle projected onto areference plane, said reference plane being defined by a longitudinalreference line parallel to the navigable element and orientatedperpendicularly to a surface of the navigable element;

obtaining sensor data by scanning the environment around the vehicleusing at least one sensor;

determining real time scan data using the sensor data, wherein the realtime scan data comprises an outline of objects in the environment aroundthe vehicle projected onto the reference plane as determined from thesensor data;

calculating a correlation between the localisation reference data andthe real time scan data to determine a longitudinal alignment offset;and

using the determined alignment offset to adjust the deemed currentposition to determine the longitudinal position of the vehicle relativeto the digital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

The location reference data may be stored in association with thedigital map, e.g. associated with the relevant navigable element(s),such that the outline of objects in the environment around the vehicleprojected onto the reference plane is already determined. In otherembodiments, however, the location reference data can be stored in adifferent format, and the stored data is processed so as to determinethe outline. For example, in embodiments, as in the earlier describedaspects of the invention, the location reference data comprises one ormore depth maps, e.g. raster images, each depth map representing thelateral distance to surfaces in the environment for a plurality oflongitudinal positions and elevations. The depth maps may be inaccordance with any of the earlier aspects and embodiments. In otherwords, the location reference data comprises at least one depth map,e.g. raster image, indicative of the environment around the vehicle,wherein each pixel of the at least one depth map is associated with aposition in the reference plane, and each pixel includes a channelrepresenting the lateral distance, e.g. normal to the reference plane,to a surface of an object in the environment. In such embodiments, therelevant depth map, e.g. raster image, is processed using an edgedetection algorithm to generate the outline of the objects in theenvironment. The edge detection algorithm can include a Canny operator,a Prewitt operator and the like. In preferred embodiments, however, theedge detection is performed using a Sobel operator. The edge detectionoperator can be applied in both the height (or elevation) andlongitudinal domains, or in just one of said domains. For example, in apreferred embodiment, the edge detection operator is applied only in thelongitudinal domain.

Similarly, the outline of objects in the environment around the vehicleprojected onto the reference plane can be determined directly from thesensor data obtained by the at least one sensor. Alternatively, in otherembodiments, the sensor data can be used to determine one or more depthmaps, e.g. raster images, each depth map representing the lateraldistance to surfaces in the environment for a plurality of longitudinalpositions and elevations. In other words, the real time scan datacomprises at least one depth map, e.g. raster image, indicative of theenvironment around the vehicle, wherein each pixel of the at least onedepth map is associated with a position in the reference plane, and eachpixel includes a channel representing the lateral distance, e.g. normalto the reference plane, to a surface of an object in the environment asdetermined using the at least one sensor. The relevant depth map, e.g.raster image, can then be processed using an edge detection algorithm,preferably the same edge detection algorithm applied to the locationreference data, to determine the outline of the real time scan data. Theedge detection operator can be applied in both the height (or elevation)and longitudinal domains, or in just one of said domains. For example,in a preferred embodiment, the edge detection operator is applied onlyin the longitudinal domain.

In embodiments, a blurring operator is applied to the outline of atleast one of the localisation reference data and the real time scandata, before the two sets of data are correlated. The blurring operationcan be applied in both the height (or elevation) and longitudinaldomains, or in just one of said domains. For example, in a preferredembodiment, the blurring operator is applied only in the height domain.The blurring operation can take into account any tilting of the vehiclewhile obtaining the real time scan data and/or the localisationreference data, such that, for example, the outline is shifted slightlyupwards or downwards in the height domain.

In accordance with the invention of any of its aspects or embodiments,the deemed current e.g. longitudinal position of the vehicle can beobtained, at least initially, from an absolute positioning system, suchas a satellite navigation device, such as GPS, GLONASS, the EuropeanGalileo positioning system, COMPASS positioning system or IRNSS (IndianRegional Navigational Satellite System). It will be appreciated,however, that other location determining means can be used, such asusing mobile telecommunications, surface beacons or the like.

The digital map may comprise a three dimensional vector modelrepresenting the navigable elements of the navigable network, e.g. roadsof a road network, in which each lane of the navigable elements, e.g.roads, are represented separately. Thus, a lateral position of thevehicle on the road may be known by determining the lane in which thevehicle is travelling, e.g. through image processing of a camera mountedto the vehicle. In such embodiments, a longitudinal reference line canbe, for example, an edge or boundary of a lane of the navigable elementor a centre line of a lane of the navigable element.

The real time scan data may be obtained on a left side of the vehicleand a right side of the vehicle. This helps to reduce the effect oftransient features on the position estimation. Such transient featuresmay be, for example, parked vehicles, overtaking vehicles or vehiclestravelling the same route in the opposite direction. Thus, real timescan data can record features present on both sides of the vehicle. Insome embodiments, the real time scan data may be obtained from either aleft side of the vehicle or a right side of the vehicle.

In embodiments in which the localisation reference data and the realtime scan data are each in respect of left and right sides of thevehicle, the comparison of the real time scan data from the left side ofthe vehicle with the localisation reference data from the left side ofthe navigable element and the comparison of the real time scan data fromthe right side of the vehicle with the localisation reference data fromthe right side of the navigable element may be a single comparison.Thus, when the scan data comprises data from the left side of thenavigable element and data from the right side of the navigable element,the scan data may be compared as a single data set, significantlyreducing the processing requirements compared to where the comparisonfor the left side of the navigable element and the comparison for theright side of the navigable element are performed separately.

Regardless of whether it relates to the left and right sides of avehicle, comparing the real time scan data to the localisation referencedata may comprise calculating a cross-correlation, preferably anormalised cross-correlation, between the real time scan data and thelocalisation reference data. The method may comprise determining thepositions at which the data sets are most aligned. Preferably thealignment offset between the depth maps that is determined is at least alongitudinal alignment offset, and the positions at which the data setsare most aligned are longitudinal positions. The step of determining thelongitudinal positions at which the data sets are most aligned maycomprise longitudinally shifting the depth map, e.g. raster image,provided by the depth map based on the real time scan data and the depthmap, e.g. raster image, provided by the depth map based on thelocalisation reference data relative to one another until the depth mapsare aligned. This may be performed in the image domain.

The determined longitudinal alignment offset is used to adjust thedeemed current position to adjust the longitudinal position of thevehicle relative to the digital map.

Alternatively, or preferably additionally to determining a longitudinalalignment offset between the depth maps, it is desirable to determine alateral alignment offset between the depth maps. The determined lateralalignment offset may then be used to adjust the deemed current lateralposition of the vehicle and hence to determine the position of thevehicle relative to the digital map. Preferably a longitudinal alignmentoffset is determined, which may be carried out in any of the mannersabove described, and a lateral alignment offset is additionallydetermined. The determined lateral and longitudinal alignment offsetsare then used together to adjust both the longitudinal and lateralpositions of the vehicle relative to the digital map.

The method may comprise determining a longitudinal alignment offsetbetween the depth maps, e.g. by calculating a correlation between thelocalisation reference data and the real time scan data, and may furthercomprise: determining a lateral offset between the depth maps; and usingthe determined lateral and longitudinal alignment offsets to adjust thedeemed current position to determine the position of the vehiclerelative to the digital map.

The longitudinal alignment offset is preferably determined before thelateral alignment offset. In accordance with certain embodimentsdescribed below, the lateral alignment offset may be determined basedupon first determining a longitudinal offset between the depth maps, andlongitudinally aligning the depth maps relative to one another based onthe offset.

The lateral offset is preferably determined based on the most common,i.e. mode lateral offset between corresponding pixels of the depth maps.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of navigable elements ofa navigable network along which the vehicle is travelling, the methodcomprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line associated with the navigable element, each pixel ofthe at least one depth map being associated with a position in thereference plane associated with the navigable element along which thevehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, wherein the real time scan datacomprises at least one depth map indicative of an environment around thevehicle, each pixel of the at least one depth map being associated witha position in the reference plane associated with the navigable element,and the pixel including a depth channel representing the distance to asurface of an object in the environment along the predetermineddirection from the associated position of the pixel in the referenceplane as determined using the at least one sensor;

determining a longitudinal alignment offset between the depth maps ofthe localisation reference data and the real time scan data bycalculating a correlation between the localisation reference data andthe real time scan data;

determining a lateral alignment offset between the depth maps, whereinthe lateral offset is based on a most common lateral offset betweencorresponding pixels of the depth maps; and

using the determined longitudinal and lateral alignment offsets toadjust the deemed current position to determine the position of thevehicle relative to the digital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with these aspects and embodiments of the invention inwhich a lateral alignment offset is determined, the most common lateralalignment offset may be determined by consideration of the depth channeldata of the corresponding pixels of the depth map. The most commonlateral alignment offset is based upon the determined lateral alignmentoffsets determined between respective pairs of correspondinglypositioned pixels of the depth maps, and preferably is based upon thelateral alignment offsets of each pair of corresponding pixels. In orderto determine the lateral alignment offset between corresponding pixelsof the depth maps, the corresponding pairs of pixels in the depth mapsmust be identified. The method may comprise identifying correspondingpairs of pixels in the depth maps. Preferably the longitudinal alignmentoffset is determined before the lateral alignment offset. The depth mapsare desirably shifted relative to one another until they arelongitudinally aligned to enable the corresponding pixels in each depthmap to be identified.

The method may therefore further comprise aligning the depth mapslongitudinally relative to one another based on the determinedlongitudinal alignment offset. The step of aligning the depth mapslongitudinally with one another may comprise shifting either or both ofthe depth maps longitudinally. The shifting of the depth mapslongitudinally relative to one another may be carried out in the imagedomain. Thus the step of aligning the depth maps may comprise shiftingthe raster images corresponding to each depth map longitudinallyrelative to one another. The method may further comprise cropping thesize of the image provided by the localisation reference data depth mapto correspond to the size of the image provided by the real time scandata depth map. This may facilitate comparison between the depth maps.

Once the corresponding pixels in the two depth maps have beenidentified, a lateral offset between each pair of corresponding pixelsmay be determined. This may be achieved by comparison of the distance tothe surface of an object in the environment along the predetermineddirection from the position of the pixel in the reference planeindicated by the depth channel data associated with each pixel. Asdescribed earlier, the depth map preferably has a variable depthresolution. The lateral alignment offset between each pair ofcorresponding pixels may be based on the difference in the distancesindicated by the depth channel data of the pixels. The method maycomprise using a histogram to identify a most common lateral alignmentoffset between the corresponding pixels of the depth maps. The histogrammay be indicative of the frequency of occurrence of different lateralalignment offsets between corresponding pairs of pixels. The histogrammay be indicative of a probability density function of the lateralalignment shift, with the mode reflecting the most probable shift.

In some embodiments, each pixel has a colour indicative of the value ofthe depth channel of the pixel. The comparison of the depth values ofthe corresponding pixels may then comprise comparing the colours of thecorresponding pixels of the depth map. The difference in colour betweenthe corresponding pixels may be indicative of the lateral alignmentoffset between the pixels, e.g. when the depth map has a fixed depthresolution.

In these embodiments in which a lateral alignment offset is determined,the current longitudinal and lateral positions of the vehicle relativeto the digital map may be adjusted.

In accordance with any of the aspects or embodiments of the invention inwhich the current position of the vehicle is adjusted, whether thelongitudinal and/or lateral position, the current position that isadjusted may be an estimation of the current position obtained in anysuitable manner, such as from an absolute position determining system orother location determining system, as described above. For example, GPS,or dead reckoning may be used. As will be appreciated, the absoluteposition is preferably matched to the digital map to determine aninitial position relative to the digital map; the longitudinal and/orlateral corrections are then applied to the initial position to improvethe position relative to the digital map.

The Applicant has realised that while the above described techniques maybe useful in adjusting a position of a vehicle with respect to a digitalmap, they will not correct the heading of the vehicle. In preferredembodiments the method further comprises using the localisationreference data and real time scan data depth maps to adjust a deemedheading of the vehicle. This further step is preferably carried out inaddition to determining longitudinal and lateral alignment offsets ofthe depth maps in accordance with any of the embodiments describedabove. In these embodiments, the deemed heading of the vehicle may bedetermined in any suitable manner, e.g. using GPS data etc., asdescribed in relation to determining the deemed position of a vehicle.

It has been found that where the deemed heading of the vehicle isincorrect, the lateral alignment offset between the corresponding pixelsof the depth maps will vary in the longitudinal direction along thedepth maps i.e. along the depth map images. It has been found that aheading offset may be determined based upon a function indicative of avariation in lateral alignment offset between corresponding pixels ofthe depth maps with respect to longitudinal position along the depthmaps. The step of determining the heading offset may incorporate any ofthe features described earlier in relation to determining the lateralalignment offset of corresponding pixels. Thus, the method preferablyfirst comprises shifting the depth maps relative to one another tolongitudinally align the depth maps.

The method may therefore further comprise: determining a longitudinalalignment offset between the depth maps; determining a functionindicative of a variation in the lateral alignment offset between thecorresponding pixels of the depth maps with respect to longitudinalposition of the pixels along the depth maps; and using the determinedfunction to adjust the deemed current heading of the vehicle todetermine the heading of the vehicle relative to the digital map.

The determined lateral alignment offset of between corresponding pixelsis, as described above, preferably based on the difference in the valuesindicated by the depth channel data of the pixels, e.g. by reference tothe colours of the pixels.

In these aspects or embodiments, the determined function is indicativeof a heading offset of the vehicle.

The step of determining the function indicative of the variation inlateral alignment offset with longitudinal position may comprisedetermining an average (i.e. mean) lateral alignment offset ofcorresponding pixels of the depth maps in each of a plurality ofvertical sections through the depth maps along the longitudinaldirection of the depth maps. The function may then be obtained basedupon the variation in the average lateral alignment offset determinedfor each vertical section along the longitudinal direction of the depthmaps. It will be appreciated that at least some, and optionally each, ofthe corresponding pairs of pixels in the depth maps are considered indetermining the function.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of navigable elements ofa navigable network along which the vehicle is travelling, the methodcomprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line associated with the navigable element, each pixel ofthe at least one depth map being associated with a position in thereference plane associated with the navigable element along which thevehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, wherein the real time scan datacomprises at least one depth map indicative of an environment around thevehicle, each pixel of the at least one depth map being associated witha position in the reference plane associated with the navigable element,and the pixel including a depth channel representing the distance to asurface of an object in the environment along the predetermineddirection from the associated position of the pixel in the referenceplane as determined using the at least one sensor;

determining a function indicative of a variation in a lateral alignmentoffset between corresponding pixels of the localisation reference andreal time sensor data depth maps with respect to longitudinal positionof the pixels along the depth maps; and

using the determined function to adjust the deemed current heading ofthe vehicle to determine the heading of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In these aspects and embodiments of the invention, further steps may betaken to improve the determined heading offset, e.g. by filtering outnoisy pixels, or weighting the mean pixel depth difference values withinlongitudinal sections of the depth map or image by reference to thenumber of significant pixels taken into account in the section.

As mentioned above, the depth maps of the localisation reference data,and thus also of the real-time data, can be transformed so as to alwaysbe associated with a linear reference line. Due to this linearisation ofthe depth maps, when the navigable element is curved, it has be foundthat it is not possible to directly apply the determined longitudinal,lateral and/or heading corrections. The Applicant has identified that acomputationally efficient method of adjusting or correcting the currentposition of the vehicle relative to the digital map involves applyingeach of the corrections in a series of incremental, independent linearupdate steps.

Thus in preferred embodiments, a determined longitudinal offset isapplied to the current position of the vehicle relative to the digitalmap, and the at least one depth map of the real time scan data isrecomputed based on the adjusted position. A lateral offset, determinedusing the recomputed real time scan data, is then applied to theadjusted position of the vehicle relative to the digital map, and the atleast one depth map of the real time scan data is recomputed based onthe further adjusted position. A skew, i.e., heading offset, determinedusing the recomputed real time scan data, is then applied to the furtheradjusted position of the vehicle relative to the digital map, and the atleast one depth map of the real time scan data is recomputed based onthe again adjusted position. These steps are preferably repeated for anynumber of times as needed, until such a time that there is zero, orsubstantially zero, longitudinal offset, lateral offset and skew.

It will be appreciated that the generated localisation reference dataobtained in accordance with the invention in any of its aspects orembodiments may be used in other manners together with real time scandata to determine a more accurate position of a vehicle, or indeed, forother purposes. In particular, the Applicant has recognised that it maynot always be possible, or at least convenient, to determine acorresponding depth map using real time scan data for comparison to thedepth map of the localisation reference scan data. In other words, itmay not be appropriate to carry out comparison of the datasets in theimage domain. This may be the case in particular where the types ofsensor available on the vehicle differ from those used to obtain thelocalisation reference data.

In accordance with some further aspects and embodiments of the inventionthe method comprises using the localisation reference data to determinea reference point cloud indicative of the environment around thenavigable element, the reference point cloud including a set of firstdata points in a three-dimensional coordinate system, wherein each firstdata point represents a surface of an object in the environment.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map, the localisation reference data providing a compressedrepresentation of an environment around at least one navigable elementof a navigable network represented by the digital map, the methodcomprising, for at least one navigable element represented by thedigital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the navigable element projected onto a reference plane, said reference plane being defined by a referenceline associated with the navigable element, each pixel of the at leastone depth map being associated with a position in the reference planeassociated with the navigable element, and the pixel including a depthchannel representing the distance to a surface of an object in theenvironment along a predetermined direction from the associated positionof the pixel in the reference plane;

associating the generated localisation reference data with the digitalmap data; and

using the localisation reference data to determine a reference pointcloud indicative of the environment around the navigable element, thereference point cloud including a set of first data points in athree-dimensional coordinate system, wherein each first data pointrepresents a surface of an object in the environment.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with a further aspect of the invention there is provided amethod of generating localisation reference data associated with adigital map representing elements of a navigable network, thelocalisation reference data providing a compressed representation of anenvironment around at least one junction of the navigable networkrepresented by the digital map, the method comprising, for at least onejunction represented by the digital map:

generating localisation reference data comprising at least one depth mapindicative of an environment around the junction projected on to areference plane, said reference plane being defined by a reference linedefined by a radius centred on a reference point associated with thejunction, each pixel of the at least one depth map being associated witha position in the reference plane associated with the junction, and thepixel including a depth channel representing the distance to a surfaceof an object in the environment along a predetermined direction from theassociated position of the pixel in the reference plane;

associating the generated localisation reference data with digital mapdata indicative of the junction; and

using the localisation reference data to determine a reference pointcloud indicative of the environment around the junction, the referencepoint cloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

The reference point cloud including a set of first data points in athree-dimensional coordinate system, wherein each first data pointrepresents a surface of an object in the environment, may be referred toherein as the “3D point cloud”. The 3D point cloud obtained inaccordance with these further aspects of the invention may be used indetermining the positioning of a vehicle.

In some embodiments the method may comprise using the generatedlocalisation reference data of the invention in any of its aspects orembodiments in determining a position of a vehicle relative to a digitalmap, the digital map comprising data representative of navigableelements of a navigable network along which the vehicle is travelling,the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementor junction of the navigable network, using the localisation referencedata to determine a reference point cloud indicative of the environmentaround the vehicle, the reference point cloud including a set of firstdata points in a three-dimensional coordinate system, wherein each firstdata point represents a surface of an object in the environment;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, the real time scan data comprising apoint cloud indicative of the environment around the vehicle, the pointcloud including a set of second data points in the three-dimensionalcoordinate system, wherein each data point represents a surface of anobject in the environment as determined using the at least one sensor;

calculating a correlation between the point cloud of the real time scandata and the point cloud of the obtained localisation reference data todetermine an alignment offset between the point clouds; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative of navigable elements ofa navigable network along which the vehicle is travelling, the methodcomprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line associated with the navigable element, each pixel ofthe at least one depth map being associated with a position in thereference plane associated with the navigable element along which thevehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

using the localisation reference data to determine a reference pointcloud indicative of the environment around the vehicle, the referencepoint cloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, the real time scan data comprising apoint cloud indicative of the environment around the vehicle, the pointcloud including a set of second data points in the three-dimensionalcoordinate system, wherein each data point represents a surface of anobject in the environment as determined using the at least one sensor;

calculating a correlation between the point cloud of the real time scandata and the point cloud of the obtained localisation reference data todetermine an alignment offset between the point clouds; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with a further aspect of the invention there is provided amethod of determining a position of a vehicle relative to a digital map,the digital map comprising data representative a junction of a navigablenetwork through which the vehicle is travelling, the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle at a junction of thenavigable network, wherein the location reference data comprises atleast one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line defined by a radius centred on a reference pointassociated with the junction, each pixel of the at least one depth mapbeing associated with a position in the reference plane associated withthe junction through which the vehicle is travelling, and the pixelincluding a depth channel representing the distance to a surface of anobject in the environment along a predetermined direction from theassociated position of the pixel in the reference plane;

using the localisation reference data to determine a reference pointcloud indicative of the environment around the vehicle, the referencepoint cloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment;

determining real time scan data by scanning the environment around thevehicle using at least one sensor, the real time scan data comprising apoint cloud indicative of the environment around the vehicle, the pointcloud including a set of second data points in the three-dimensionalcoordinate system, wherein each data point represents a surface of anobject in the environment as determined using the at least one sensor;

calculating a correlation between the point cloud of the real time scandata and the point cloud of the obtained localisation reference data todetermine an alignment offset between the point clouds; and

using the determined alignment offset to adjust the deemed currentposition to determine the position of the vehicle relative to thedigital map.

The reference point cloud in these further aspects including a set ofsecond data points in a three-dimensional coordinate system, whereineach second data point represents a surface of an object in theenvironment, may be referred to herein as a “3D point cloud”.

In these further aspects or embodiments of the invention, thelocalisation reference data is used to obtain a 3D reference pointcloud. This is indicative of the environment around the navigableelement or junction to which the data relates, and hence to theenvironment around a vehicle when travelling along the navigable elementor through the junction. The point cloud of the real time sensor datarelates to the environment around the vehicle, and may therefore also bereferred to as relating to the environment around the navigable elementor junction where the vehicle is located. In some preferred embodimentsthe 3D point cloud obtained based upon the localisation reference datais compared to a 3D point cloud indicative of the environment around thevehicle (i.e. when travelling on the relevant element or through thejunction) obtained based on the real time scan data. The position of thevehicle may then be adjusted based on this comparison, rather than acomparison of depth maps, e.g. raster images.

The real time scan data point cloud is obtained using one or moresensors associated with the vehicle. A single sensor, or multiple suchsensors may be used, and in the latter case, any combination of sensortypes may be used. The sensors may include any one or ones of; a set ofone or more laser scanners, a set of one or more radar scanners, and aset of one or more cameras e.g. a single camera or a pair of stereocameras. A single laser scanner, radar scanner and/or camera may beused. Where the vehicle is associated with a camera or cameras, imagesobtained from the one or more cameras may be used to construct a threedimensional scene indicative of the environment around the vehicle, andthe 3 dimensional point cloud may be obtained using the threedimensional scene. For example, where the vehicle uses a single camera,a point cloud may be determined therefrom by obtaining a sequence of twodimensional images from the camera as the vehicle travels along thenavigable element or through the junction, using the sequence of twodimensional images to construct a three dimensional scene, and using thethree dimensional scene to obtain the three dimensional point cloud.Where the vehicle is associated with stereo cameras, the images obtainedfrom the cameras may be used to obtain a three dimensional scene whichis then used to obtain the three dimensional point cloud.

By transforming the depth map of the localisation reference data into a3D point cloud, it may be compared to a 3D point cloud obtained throughthe real time scanning using the vehicle sensor(s), regardless of whatthey may be. For example, the localisation reference data may be basedupon a reference scan using a variety of sensor types, including a laserscanner, camera and radar scanner. A vehicle may or may not have acorresponding set of sensors. For example, typically a vehicle may onlyinclude one or more cameras.

The localisation reference data may be used to determine a referencepoint cloud indicative of the environment around the vehicle whichcorresponds to a point cloud that would be expected to be generated bythe at least one sensor of the vehicle. Where the reference point cloudwas obtained using the same type of sensors as those of the vehicle,this may be straightforward, and all of the localisation reference datamay be used in constructing the 3D reference point cloud. Similarly,under certain conditions, data sensed by one type of sensor may besimilar to that sensed by another. For example, objects sensed by alaser sensor in providing the reference localisation data may beexpected to also be sensed by a camera of a vehicle during daylight.However, the method may comprise including only those points in the 3Dpoint cloud that would be expected to be detected by a sensor or sensorsof the type associated with the vehicle and/or expected to be detectedunder current conditions. The localisation reference data may comprisedata which enables an appropriate reference point cloud to be generated.

In some embodiments, as described above, each pixel of the localisationreference data further includes at least one channel indicative of avalue of a sensed reflectivity. Each pixel may include one or more of; achannel indicative of a value of a sensed laser reflectivity, and achannel indicative of a value of a sensed radar reflectivity. Preferablychannels indicative of both radar and laser reflectivity are provided.The step of generating the 3D point cloud based on the localisationreference data is then preferably performed using the sensedreflectivity data. The generation of the 3D point cloud may also bebased upon the type of sensor or sensors of the vehicle. The method maycomprise using the reflectivity data and data indicative of the type ofsensor or sensors of the vehicle to select 3D points for inclusion inthe reference 3D point cloud. The data of the reflectivity channel isused to select data from the depth channel for use in generating the 3Dpoint cloud. The reflectivity channel gives an indication of whether aparticular object would be sensed by the relevant sensor type (whereappropriate, under current conditions).

For example, where the reference data is based upon data obtained from alaser scanner and a radar scanner, and the vehicle has only a radarscanner, the radar reflectivity value may be used to select those pointsfor inclusion in the 3D point cloud that would be expected to be sensedby the radar scanner of the vehicle. In some embodiments each pixelincludes a channel indicative of a radar reflectivity, and the methodcomprises the step of using the radar reflectivity data to generate a 3Dreference point cloud that contains only those points that would besensed by a radar sensor. Where the method further comprises comparingthe 3D reference point cloud to a 3D point cloud obtained based on thereal time scan data, the 3D point cloud of the real time scan data isthen based on data obtained from a radar scanner. The vehicle mayinclude only a radar scanner.

While a vehicle may include radar and/or laser scanners, in many cases,a car may include only a camera or cameras. The laser reflectivity datamay provide a way of obtaining a 3D reference point cloud thatcorrelates to a 3D point cloud that would be expected to be sensed by avehicle having only a camera or cameras as sensors under conditions ofdarkness. The laser reflectivity data provides an indication of thoseobjects that might be expected to be detected by a camera during thedark. In some embodiments each pixel includes a channel indicative of alaser reflectivity, and the method comprises the step of using the laserreflectivity data to generate a 3D reference point cloud that containsonly those points that would be sensed by a camera of a vehicle duringconditions of darkness. Where the method further comprises comparing the3D reference point cloud to a 3D point cloud obtained based on the realtime scan data, the 3D point cloud of the real time scan data may thenbe based on data obtained from a camera under conditions of darkness.

It is believed that obtaining reference localisation data in the form ofa three dimensional point cloud, and using such data to reconstruct areference view, e.g. image which would be expected to be obtained fromone or more cameras of a vehicle under the applicable conditions, andwhich may then be compared to an image obtained by the camera, isadvantageous in its own right.

In some embodiments the method may comprise using the generatedlocalisation reference data of the invention in any of its aspects orembodiments in reconstructing a view that would be expected to beobtained from one or more cameras associated with a vehicle travellingalong a navigable element of a navigable network or through a junctionrepresented by a digital map under applicable conditions, the methodcomprising: obtaining localisation reference data associated with thedigital map for a deemed current position of the vehicle along anavigable element or junction of the navigable network or at thejunction, using the localisation reference data to determine a referencepoint cloud indicative of the environment around the vehicle, thereference point cloud including a set of first data points in athree-dimensional coordinate system, wherein each first data pointrepresents a surface of an object in the environment; and using thereference point cloud to reconstruct a reference view that would beexpected to be obtained by the one or more cameras associated with thevehicle when traversing the navigable element or the junction under theapplicable conditions. The method may further comprise determining areal time view of the environment around the vehicle using the one ormore cameras, and comparing the reference view to the real time viewobtained by the one or more cameras.

In accordance with a further aspect of the invention there is provided amethod of reconstructing a view that would be expected to be obtainedfrom one or more cameras associated with a vehicle travelling along anavigable element of a navigable network represented by a digital mapunder applicable conditions, the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line associated with the navigable element, each pixel ofthe at least one depth map being associated with a position in thereference plane associated with the navigable element along which thevehicle is travelling, and the pixel including a depth channelrepresenting the distance to a surface of an object in the environmentalong a predetermined direction from the associated position of thepixel in the reference plane;

using the localisation reference data to determine a reference pointcloud indicative of the environment around the vehicle, the referencepoint cloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment;

using the reference point cloud to reconstruct a reference view thatwould be expected to be obtained by the one or more cameras associatedwith the vehicle when traversing the navigable element under theapplicable conditions;

determining a real time view of the environment around the vehicle usingthe one or more cameras; and

comparing the reference view to the real time view obtained by the oneor more cameras.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

In accordance with a further aspect of the invention there is provided amethod of reconstructing a view that would be expected to be obtainedfrom one or more cameras associated with a vehicle travelling through ajunction of a navigable network represented by a digital map underapplicable conditions, the method comprising:

obtaining localisation reference data associated with the digital mapfor a deemed current position of the vehicle along a navigable elementof the navigable network, wherein the location reference data comprisesat least one depth map indicative of an environment around the vehicleprojected on to a reference plane, the reference plane being defined bya reference line defined by a radius centred on a reference pointassociated with the junction, each pixel of the at least one depth mapbeing associated with a position in the reference plane associated withthe junction through which the vehicle is travelling, and the pixelincluding a depth channel representing the distance to a surface of anobject in the environment along a predetermined direction from theassociated position of the pixel in the reference plane;

using the localisation reference data to determine a reference pointcloud indicative of the environment around the vehicle, the referencepoint cloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment;

using the reference point cloud to reconstruct a reference view thatwould be expected to be obtained by the one or more cameras associatedwith the vehicle when traversing the navigable element under theapplicable conditions;

determining a real time view of the environment around the vehicle usingthe one or more cameras; and

comparing the reference view to the real time view obtained by the oneor more cameras.

The present invention in accordance with this further aspect may includeany or all of the features described in relation to the other aspects ofthe invention, to the extent that it is not mutually inconsistenttherewith.

These aspects of the invention are particularly advantageous in allowinga reference view to be constructed that may be compared to a real timeview obtained by the camera(s) of a vehicle, but based upon localisationreference data which may be obtained from different types of sensor. Ithas been recognised that in practice, many vehicles will only beequipped with a camera or cameras, rather than more specific orsophisticated sensors, such as may be used to obtain the reference data.

In these further aspects and embodiments of the invention, the resultsof the comparison of the reference and real time views may be used asdesired. For example, the results of the comparison may be used indetermining the position of the vehicle as in the earlier describedaspects and embodiments. The method may comprise calculating acorrelation between the real time view and the reference view todetermine an alignment offset between the views; and using thedetermined alignment offset to adjust a deemed current position of thevehicle to determine the position of the vehicle relative to the digitalmap.

The applicable conditions are those conditions applicable at the currenttime, and may be a lighting condition. In some embodiments, theapplicable condition is a condition of darkness.

The reference view is reconstructed using a 3D reference point cloudthat may be obtained from the localisation reference data in accordancewith any of the embodiments described above. The step of reconstructingthe reference view that would be expected to be obtained by the one ormore cameras preferably comprises using data of a reflectivity datachannel associated with the pixels of the depth map of the localisationreference data. Preferably, therefore, each pixel of the localisationreference data further includes at least one channel indicative of avalue of a sensed laser reflectivity, and the step of generating the 3Dpoint cloud based on the localisation reference data is performed usingthe sensed laser reflectivity data. The laser reflectivity data may beused to select data from the depth channel for use in generating thereference 3D point cloud to result in the reconstructed reference viewcorresponding to a view that could be expected to be obtained from theone or more cameras of the vehicle, e.g. including those objects thatcould be intended to be seen under applicable conditions, e.g. darkness.The one or more cameras of the vehicle may be a single camera, or a pairof stereo cameras, as described above.

The comparison of the real time scan data to the localisation referencedata that may be performed in accordance with the invention in itsvarious aspects and embodiments, whether by comparison of depth maps, orthrough comparison of point clouds, or a reconstructed and real timeimage, may be performed over a window of data. The window of data is awindow of data in the direction of travel e.g. longitudinal data. Thus,windowing the data allows the comparison to take account of a subset ofthe available data. The comparison may be performed periodically foroverlapping windows. At least some overlap in the windows of data usedfor the comparison is desirable. For example, this may ensure that thedifferences between neighbouring calculated e.g. longitudinal offsetvalues are smoothed over the data. The window may have a lengthsufficient for the accuracy of the offset calculation to be invariant totransient features, preferably the length being at least 100 m. Suchtransient features may be, for example, parked vehicles, overtakingvehicles or vehicles travelling the same route in the oppositedirection. In some embodiments, the length is at least 50 m. In someembodiments, the length is 200 m. In this way, the sensed environmentdata is determined for a stretch (e.g. a longitudinal stretch) of road,the ‘window’, e.g. 200 m, and the resultant data is then compared to thelocalisation reference data for the stretch of road. By performing thecomparison over a stretch of road of this size, i.e. one that issubstantially larger than the length of the vehicle, non-stationary ortemporary objects, such as other vehicles on the road, vehicles stoppedon the side of the road, etc, will typically not impact the result ofthe comparison.

At least a portion of the localisation reference data used in accordancewith the invention in any of its aspects or embodiments may be storedremotely. Preferably, where a vehicle is involved, at least a portion ofthe localisation reference data is stored locally on the vehicle. Thus,even though the localisation reference data is available throughout theroute, it need not be continually transferred onto the vehicle and thecomparison may be performed on the vehicle.

The localisation reference data may be stored in a compressed format.The localisation reference data may have a size that corresponds to 30KB/km or less.

The localisation reference data may be stored for at least some, andpreferably all, of the navigable elements of the navigable networkrepresented in the digital map. Thus, the position of the vehicle can becontinually determined anywhere along a route travelled by a vehicle.

In embodiments, the reference localisation data may have been obtainedfrom a reference scan using at least one device located on a mobilemapping vehicle which has previously travelled along the navigableelement that is subsequently travelled by a vehicle. Thus, the referencescan may have been acquired using a different vehicle than the currentvehicle for which a position is being continually determined. In someembodiments, the mobile mapping vehicle is of a similar design to thevehicle for which the position is being continually determined.

The real time scan data and/or the reference scan data may be obtainedusing at least one range-finder sensor. The range-finder sensor may beconfigured to operate along a single axis. The range-finder sensor maybe arranged to perform a scan in a vertical axis. When the scan isperformed in the vertical axis, distance information for planes atmultiple heights is collected, and thus the resultant scan issignificantly more detailed. Alternatively, or in addition, therange-finder sensor may be arranged to perform a scan in a horizontalaxis.

The range-finder sensor may be a laser scanner. The laser scanner maycomprise a laser beam scanned across the lateral environment usingmirrors. Additionally, or alternatively, the range-finder sensor may beradar scanner and/or a pair of stereo cameras.

The invention extends to a device, e.g. navigation device, vehicle, etc,having means, such as one or more processors arranged, e.g. programmed,to perform any of the methods described herein.

The steps of generating localisation reference data described herein arepreferably performed by a server or other similar computing device.

The means for carrying out any of the steps of the method may comprise aset of one or more processors configured, e.g. programmed, for doing so.A given step may be carried out using the same or a different set ofprocessors to any other step. Any given step may be carried out using acombination of sets of processors. The system may further comprise datastorage means, such as computer memory, for storing, for example, thedigital map, the localisation reference data and/or the real time scandata.

The methods of the present invention are, in preferred embodiments,implemented by a server or similar computing device. In other words, themethods of the presented invention are preferably computer implementedmethods. Thus, in embodiments, the system of the present inventioncomprises a server or similar computing device comprising the means forcarrying out the various steps described, and the method steps describedherein are carried out by a server.

The invention further extends to a computer program product comprisingcomputer readable instructions executable to perform or cause a deviceto perform any of the methods described herein. The computer programproduct is preferably stored in a non-transitory physical storagemedium.

As will be appreciated by those skilled in the art, the aspects andembodiments of the present invention can, and preferably do, include anyone or more or all of the preferred and optional features of theinvention described herein in respect of any of the other aspects of theinvention, as appropriate.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of exampleonly, with reference to the accompanying drawings, in which:

FIG. 1 is a representation of a portion of a planning map;

FIG. 2 shows a portion of a planning map overlaid on an image of a roadnetwork;

FIGS. 3 and 4 show an exemplary mobile mapping system that can be usedto collect data for building maps;

FIG. 5 shows a 3D view of data obtained from a laser scanner, while FIG.6 shows a side view projection of the data obtained from the laserscanner;

FIG. 7 shows a vehicle, in accordance with an embodiment, travellingalong a road while sensing its surroundings;

FIG. 8 shows a comparison of localisation reference data compared tosensed environment data, e.g. as collected by the vehicle of FIG. 7;

FIG. 9 shows an exemplary format of how localisation reference data canbe stored;

FIG. 10A shows an example point cloud acquired by a range-finding sensormounted to a vehicle travelling along a road, while FIG. 10B shows thispoint cloud data having been converted into two depth maps;

FIG. 11 shows the offset determined following a normalisedcross-correlation calculation in an embodiment;

FIG. 12 shows another example of a correlation performed between a“reference” data set and a “local measurement” data set;

FIG. 13 shows an system positioned within a vehicle according to anembodiment;

FIG. 14A shows an exemplary raster image that is a portion of a stretchof localisation reference data;

FIG. 14B shows a bird's eye perspective of the data of FIG. 14A as twoseparate planes on the left and right of the road;

FIG. 15A shows the fixed longitudinal resolution and variable, e.g.non-linear, vertical and/or depth resolution of the localisationreference data and real time scan data;

FIG. 15B shows a function that maps heights above a reference line topixel Y coordinate values;

FIG. 15C shows a function that maps distances from a reference line topixel depth values;

FIG. 15D shows the fixed longitudinal pixel resolution, variablevertical pixel resolution, and variable depth value resolution in athree dimensional plot;

FIG. 16A shows an orthogonal projection on to a reference plane definedby a reference line associated with a road element;

FIG. 16B shows a side depth map obtained using an orthogonal projection;

FIG. 16C shows a non-orthogonal projection on to a reference planedefined by a reference line associated with a road element;

FIG. 16D shows a side depth map obtained using a non-orthogonalprojection;

FIG. 17 shows a multi-channel data format for a depth map;

FIG. 18 shows circular and linear reference lines which may be used inconstructing depth maps at cross-roads;

FIG. 19A shows a way in which objects may be projected on to a circulardepth map at different angular positions;

FIG. 19B shows an orthogonal projection of objects on to a referenceplane to provide a depth map;

FIG. 20A shows a reference depth map and a corresponding real time depthmap;

FIG. 20B shows a longitudinal correction derived from longitudinalcorrelation of the reference and real time depth maps;

FIG. 20C shows a lateral correction derived from histogrammingdifferences between pixel depth values for corresponding pixels in thereference and real time depth maps;

FIG. 20D shows how a longitudinal position and then a lateral positionof a vehicle on a road may be corrected;

FIG. 21A shows a set of vertical slices through corresponding portionsof a reference depth map;

FIG. 21B shows mean pixel depth differences for the vertical slicesplotted with respect to longitudinal distance of the vertical slicealong the depth map;

FIG. 22 shows an image of a curved road and a corresponding linearlyreferenced image for the road;

FIGS. 23A and 23B show a method for establishing the position of thevehicle, e.g. in a non-linear environment;

FIG. 24 shows an exemplary system in which data vehicle sensors iscorrelated with reference data to localise the vehicle relative to adigital map;

FIGS. 25A, 25B and 25C show a first example use case in which areference depth map is used to construct a 3D point cloud that is thencompared to a 3D point cloud obtained from vehicle laser sensors;

FIGS. 26A, 26B, 26C and 26D show a second example use case in which areference depth map is used to construct a 3D point cloud or view thatis then compared to a 3D scene or view obtained from multiple vehiclecameras or a single vehicle camera;

FIGS. 27A, 27B and 27C show a third example use case in whichreflectivity data of a depth map is used to construct a 3D point cloudor view that is then compared to a 3D scene or view obtained from avehicle camera;

FIGS. 28A and 28B show a fourth example use case in which radar data ofa depth map is used to construct a 3D point cloud that is then comparedto a 3D scene obtained using vehicle radar;

FIG. 29 shows different coordinate systems that are used in embodimentsof the invention;

FIG. 30 depicts the steps performed when correlating vehicle sensor datato reference data in order to determine the position a vehicle;

FIG. 31 illustrates the steps performed to determine the laser pointcloud in the method of FIG. 30;

FIG. 32A illustrates a first exemplary method for performing thecorrelation step in the method of FIG. 30; and

FIG. 32B illustrates a second exemplary method for performing thecorrelation step in the method of FIG. 30.

DETAILED DESCRIPTION OF THE FIGURES

It has been recognised that an improved method for determining theposition of a device, such as a vehicle, relative to a digital map(representative of a navigable network, e.g. road network) is required.In particular, it is required that the longitudinal position of thedevice relative to the digital map can be accurately determined, e.g. tosub-metre accuracy. The term “longitudinal” in this application refersto the direction along the portion of a navigable network on which thedevice, e.g. vehicle, is moving; in other words along the length of theroad on which the vehicle is travelling. The term “lateral” in thisapplication takes its normal meaning as being perpendicular to thelongitudinal direction, and thus refers to the direction along the widthof the road.

As will be appreciated, when the digital map comprises a planning map asdescribed above, e.g. a three dimensional vector model with each lane ofa road being representative separately (in contrast to relative to acentre line for the road as in standard maps), the lateral position ofthe device, e.g. vehicle, simply involves determining the lane in whichthe device is currently travelling. Various techniques are known forperforming such a determination. For example, the determination can bemade only using information obtained from the global navigationsatellite system (GNSS) receiver. Additionally or alternatively,information from a camera, laser or other imaging sensor associated withthe device can be used; for example substantial research has beencarried out in recent years, in which image data from one or more videocameras mounted within a vehicle is analysed, e.g. using various imageprocessing techniques, to detect and track the lane in which the vehicleis travelling. One exemplary technique is set out in the paper“Multi-lane detection in urban driving environments using conditionalrandom fields” authored by Junhwa Hur, Seung-Nam Kang, and Seung-WooSeo. published in the proceedings of the Intelligent Vehicles Symposium,page 1297-1302. IEEE, (2013). Here, the device may be provided with adata feed from a video camera, radar and/or LIDAR sensor and anappropriate algorithm is used to process the received data in real-timeto determine a current lane of the device or the vehicle in which thedevice is travelling. Alternatively, another device or apparatus, suchas a Mobileye system available from Mobileye N.V. may provide thedetermination of the current lane of the vehicle on the basis of thesedata feeds and then feed the determination of the current lane to thedevice, for example by a wired connection or a Bluetooth connection.

In embodiments, the longitudinal position of the vehicle can bedetermined by comparing a real-time scan of the environment around thevehicle, and preferably on one or both sides of the vehicle, to areference scan of the environment that is associated with the digitalmap. From this comparison, a longitudinal offset, if any, can bedetermined, and the position of the vehicle matched to the digital mapusing the determined offset. The position of the vehicle relative to thedigital map can therefore always be known to a high degree of accuracy.

The real-time scan of the environment around the vehicle can be obtainedusing at least one range-finder sensor that are positioned on thevehicle. The at least one range-finder sensor can take any suitableform, but in preferred embodiments comprises a laser scanner, i.e. aLIDAR device. The laser scanner can be configured to scan a laser beamacross the environment and to create a point cloud representation of theenvironment; each point indicating the position of a surface of anobject from which the laser is reflected. As will be appreciated, thelaser scanner is configured to record the time it takes for the laserbeam to return to the scanner after being reflected from the surface ofan object, and the recorded time can then be used to determine thedistance to each point. In preferred embodiments, the range-findersensor is configured to operate along a single axis so as to obtain datawithin a certain acquisition angle, e.g. between 50-90°, such as 70°;for example when the sensor comprises a laser scanner the laser beam isscanned using mirrors within the device.

An embodiment is shown in FIG. 7 in which a vehicle 100 is travellingalong a road. The vehicle is equipped with a range-finder sensor 101,102 positioned on each side of the vehicle. While a sensor is shown oneach side of the vehicle, in other embodiments only a single sensor canbe used on one side of the vehicle. Preferably, the sensors are suitablyaligned such that the data from each sensor can be combined, as isdiscussed in more detail below.

WO 2011/146523 A2 provides examples of scanners which may be used on avehicle for capturing reference data in the form of a 3dimensional pointcloud, or which could also be used on an autonomous vehicle to obtainreal time data relating to the surrounding environment.

As discussed above, the range-finder sensor(s) can be arranged tooperate along a single axis. In one embodiment, the sensor can bearranged to perform a scan in a horizontal direction, i.e. in a planeparallel to the surface of the road. This is shown, for example, in FIG.7. By continually scanning the environment as the vehicle travels alongthe road, sensed environment data as shown in FIG. 8 can be collected.The data 200 is the data collected from the left sensor 102, and showsthe object 104. The data 202 is the data collected from the right sensor101, and shows the objects 106 and 108. In other embodiments, the sensorcan be arranged to perform a scan in a vertical direction, i.e. in aplane perpendicular to the surface of the road. By continually scanningthe environment as the vehicle travels along the road, it is possible tocollect environment data in the manner of FIG. 6. As will beappreciated, by performing a scan in the vertical direction, distanceinformation for planes at multiple heights is collected, and thus theresultant scan is significantly more detailed. It will of course beappreciated that the scan could be performed along any axis as desired.

The reference scan of the environment is obtained from one or morevehicles that have previously travelled along the road, and which isthen appropriately aligned and associated with the digital map. Thereference scans are stored in a database, which is associated with thedigital map, and are referred to herein as localisation reference data.The combination of the localisation reference data when matched to adigital map can be referred to as a localisation map. As will beappreciated, the localisation map will be created remotely from thevehicles; typically by a digital map making company such as TomTomInternational B.V. or HERE, a Nokia company.

The reference scans can be obtained from specialist vehicles, such asmobile mapping vehicles, e.g. as shown in FIG. 3. In preferredembodiments, however, the reference scans can be determined from thesensed environment data that is collected by vehicles as they travelalong the navigable network. This sensed environment data can be storedand periodically sent to the digital mapping company to create, maintainand update the localisation map.

The localisation reference data is preferably stored locally at thevehicle, although it will be appreciated that the data could be storedremotely. In embodiments, and particularly when the localisationreference data is stored locally, the data is stored in a compressedformat.

In embodiments, localisation reference data is collected for each sideof a road in the road network. In such embodiments, the reference datafor each side of the road can be stored separately, or alternativelythey can be stored together in a combined data set.

In embodiments, the localisation reference data can be stored as imagedata. The image data can be colour, e.g. RGB, images, or greyscaleimages.

FIG. 9 shows an exemplary format of how the localisation reference datacan be stored. In this embodiment, the reference data for the left sideof the road is provided on the left side of the image, and the referencedata for the right side of the road is provided on the right side of theimage; the data sets being aligned such that the left-side referencedata set for a particular longitudinal position is shown opposite theright-side reference data set for the same longitudinal position.

In the image of FIG. 9, and for illustrative purposes only, thelongitudinal pixel size is 0.5 m, there are 40 pixels on each side ofthe centreline. It has also been determined that the image can be storedas grayscale images, rather than the colour (RGB) images. By storingimages in this format, the localisation reference data has a size thatcorresponds to 30 KB/km.

A further example can be seen in FIGS. 10A and 10B. FIG. 10A shows anexample point cloud acquired by a range-finding sensor mounted to avehicle travelling along a road. In FIG. 10B, this point cloud data hasbeen converted into two depth maps; one for the left side of the vehicleand the other for the right side of the vehicle, which have been placednext to each to form a composite image.

As discussed above, the sensed environment data determined by a vehicleis compared to the localisation reference data to determine if there isan offset. Any determined offset can then be used to adjust the positionof the vehicle such that it accurately matched to the correct positionon the digital map. This determined offset is referred to herein as acorrelation index.

In embodiments, the sensed environment data is determined for alongitudinal stretch of road, e.g. 200 m, and the resultant data, e.g.image data, then compared to the localisation reference data for thestretch of road. By performing the comparison over a stretch of road ofthis size, i.e. one that is substantially larger than the length of thevehicle, non-stationary or temporary objects, such as other vehicles onthe road, vehicles stopped on the side of the road, etc, will typicallynot impact the result of the comparison.

The comparison is preferably performed by calculating across-correlation between the sensed environment data and thelocalisation reference data, so as to determine the longitudinalpositions at which the data sets are most aligned. The differencebetween the longitudinal positions of both data sets at maximumalignment allows the longitudinal offset to be determined. This can beseen, for example, by the offset indicated between the sensedenvironment data and localisation reference data of FIG. 8.

In embodiments, when the data sets are provided as images, thecross-correlation comprises a normalised cross-correlation operation,such that differences in brightness, lighting conditions, etc betweenthe localisation reference data and the sensed environment data can bemitigated. Preferably, the comparison is performed periodically foroverlapping windows, e.g. of 200 m lengths, so that any offset iscontinually determined as the vehicle travels along the road. FIG. 11shows the offset determined, in an exemplary embodiment, following anormalised cross-correlation calculation between the depictedlocalisation reference data and the depicted sensed environment data.

FIG. 12 illustrates a further example of a correlation performed betweena “reference” data set and a “local measurement” data set (that isacquired by a vehicle as it travels along a road). The result of thecorrelation between the two images can be seen in the graph of “shift”against “longitudinal correlation index”, wherein the position of thelargest peak is used to determine the illustrated best-fit shift, whichcan then be used to adjust the longitudinal position of the vehiclerelative to the digital map.

As can be seen from FIGS. 9, 10B, 11 and 12, the localisation referencedata and the sensed environment data preferably are in the form of depthmaps, wherein each element (e.g. pixel when the depth map is stored asan image) comprises: a first value indicative of a longitudinal position(along a road); a second value indicative of an elevation (i.e. a heightabove ground); and a third value indicative of a lateral position(across a road). Each element, e.g. pixel, of the depth map thereforeeffectively corresponds to a portion of a surface of the environmentaround the vehicle. As will be appreciated, the size of the surfacerepresented by each element, e.g. pixel, will change with the amount ofcompression, such that an element, e.g. pixel, will represent a largersurface area with a higher level of compression of the depth map (orimage).

In embodiments, wherein the localisation reference data is stored in adata storage means, e.g. memory, of the device, the comparison step canbe performed on one or more processors within the vehicle. In otherembodiments, wherein the localisation reference data is stored remotelyfrom the vehicle, the sensed environment data can be sent to a serverover a wireless connection, e.g. via the mobile telecommunicationsnetwork. The server, which has access to the localisation referencedata, would then return any determined offset back to the vehicle, e.g.again using the mobile telecommunications network.

An exemplary system, according to an embodiment of the invention, thatis positioned within a vehicle is depicted in FIG. 13. In this system, aprocessing device referred to as a correlation index provider unitreceives a data feed from a range-finding sensor positioned to detectthe environment on the left side of the vehicle and a range-findingsensor positioned to detect the environment on the right side of thevehicle. The processing device also has access to a digital map (that ispreferably in the form of a planning map) and a database of locationreference data, which is suitably matched to the digital map. Theprocessing device is arranged to perform the method described above, andthus to compare the data feed from the range-finding sensors, optionallyafter converting the data feed into a suitable form, e.g. an image datacombining the data from both sensors, to localisation reference data todetermine a longitudinal offset and thus accurate position the vehiclerelative to the digital map. The system also comprises a horizonprovider unit, and which uses the determined position of the vehicle anddata within the digital map to provide information (referred to as a“horizon data”) concerning the upcoming portion of the navigable networkthat the vehicle is about to traverse. This horizon data can then beused to control one or more systems within the vehicle to performvarious assisted or automated driving operations, e.g. adaptive cruisecontrol, automatic lane changing, emergency brake assistance, etc.

In summary, the invention relates, at least in preferred embodiments, toa positioning method based on longitudinal correlation. The 3D spacearound a vehicle is represented in the form of two depth maps, coveringboth the left and right sides of the road, and which may be combinedinto a single image. Reference images stored in a digital map arecross-correlated with the depth maps derived from lasers or otherrange-finding sensors of the vehicle to position the vehicle preciselyalong (i.e. longitudinally) the representation of the road in thedigital map. The depth information can then be used, in embodiments, toposition the car across (i.e. laterally) the road.

In a preferred implementation, the 3D space around a vehicle isprojected to two grids parallel to road trajectory and the values ofprojections are averaged within each cell of the grid. A pixel of thelongitudinal correlator depth map has dimensions of about 50 cm alongthe driving direction and about 20 cm height. The depth, coded by pixelvalue, is quantized with about 10 cm. Although the depth map imageresolution along the driving direction is 50 cm, the resolution ofpositioning is much higher. The cross-correlated images represent a gridin which the laser points are distributed and averaged. Properup-sampling enables finding shift vectors of sub-pixel coefficients.Similarly, the depth quantization of about 10 cm does not imply 10 cmprecision of positioning across the road as the quantization error isaveraged over all of the correlated pixels. In practice, therefore, theprecision of positioning is limited mostly by laser precision andcalibration, with only very little contribution from quantization errorof longitudinal correlator index.

Accordingly, it will be appreciated, that the positioning information,e.g. the depth maps (or images), is always available (even if no sharpobjects are available in the surroundings), compact (storing wholeworld's road network is possible), and enables precision comparable oreven better than other approaches (due to its availability at any placeand therefore high error averaging potential).

FIG. 14A shows an exemplary raster image that is a portion of a stretchof location reference data. The raster image is formed by the orthogonalprojection of collected 3D laser point data onto a hyperplane defined bya reference line and oriented perpendicular to the road surface. Due tothe orthogonality of projection any height information is independent onthe distance from the reference line. The reference line itselftypically runs parallel to the lane/road boundaries. The actualrepresentation of the hyperplane is a raster format with has fixedhorizontal resolution and non-linear vertical resolution. This methodaims to maximize the information density on those heights which areimportant for detection by vehicle sensors. Experiments have shown that5-10 m of height of the raster plane is sufficient to capture enoughrelevant information necessary for later use in vehicle localization.Each individual pixel in the raster reflects a group of lasermeasurements. Just like for the vertical resolution, the resolution inthe depth information is also represented in a non-linear way, buttypically stored in 8 bit values (i.e. as a value from 0 to 255). FIG.14A shows the data for both sides of the road. FIG. 14B shows a bird'seye perspective of the data of FIG. 14A as two separate planes on theleft and right of the road.

As discussed above, vehicles equipped with front or side-mountedhorizontally mounted laser scanner sensors are able to generate, in realtime, 2D planes similar to those of the localisation reference data.Localisation of the vehicle relative to the digital map is achieved bythe correlation in image space of the a priori mapped data with thereal-time sensed and processed data. Longitudinal vehicle localisationis obtained by applying an average non-negative normalizedcross-correlation (NCC) operation calculated in overlapping movingwindows on images with 1 pixel blur in the height domain and a Sobeloperator in the longitudinal domain.

FIG. 15A shows the fixed longitudinal resolution and variable, e.g.non-linear, vertical and/or depth resolution of the localisationreference data and real time scan data. Thus, while the longitudinaldistance represented by the value a, b and c is the same, the heightrange represented by the values D, E and F are different. In particular,the height range represented by D is less than that represented by E,and the height range represented by E is less than that represented byF. Similarly, depth range represented by the value 0, i.e. surfacesclosest to the vehicle, is less than that represented by the value 100,and the depth range represented by the value 100 is less than thatrepresented by the value 255, i.e. surfaces furthest away from thevehicle. For example, the value 0 can represent 1 cm in depth, while thevalue 255 can represent 10 cm in depth.

FIG. 15B illustrates how the vertical resolution may vary. In thisexample, the vertical resolution varies based on a non-linear functionthat maps heights above the reference line to pixel Y coordinate values.As is shown in FIG. 15B, pixels closer to the reference line, which isat Y equals 40 in this example, represent lower heights. As is alsoshown in FIG. 15B, the vertical resolution is greater closer to thereference line, i.e. the change in height with respect to pixel positionis lesser for pixels which are closer to the reference line and isgreater for pixels which are further from the reference line.

FIG. 15C illustrates how the depth resolution may vary. In this example,the depth resolution varies based on a non-linear function that mapsdistances from the reference line to pixel depth (colour) values. As isshown in FIG. 15C, lower pixel depth values represent shorter distancesfrom the reference line. As is also shown in FIG. 15C, the depthresolution is greater at lower pixel depth values, i.e. the change indistance with respect to pixel depth value is lesser for lower pixeldepth values and is greater for higher pixel depth values.

FIG. 15D illustrates how a subset of pixels may map to distances alongthe reference line. As is shown in FIG. 15D, each pixel along thereference line is the same width such that the longitudinal pixelresolution is fixed. FIG. 15D also illustrates how the subset of pixelsmay map to heights above the reference line. As is shown in FIG. 15D,the pixels become progressively wider at greater distances away from thereference line, such that the vertical pixel resolution is lower atgreater heights above the reference line. FIG. 15D also illustrates howa subset of pixel depth values may map to distances from the referenceline. As is shown in FIG. 15D, the distances covered by the pixel depthvalues become progressively wider at greater distances away from thereference line, such that the depth resolution is lower at depthdistances further from the reference line.

Some further embodiments and features of the invention will now bedescribed.

As described in relation to FIG. 14A, a depth map, e.g. raster image, ofthe localisation reference data, may be provided by an orthogonalprojection on to a reference plane defined by a reference lineassociated with a road element. FIG. 16A illustrates the result of usingsuch a projection. The reference plane is perpendicular to the roadreference line shown. Here, although the height information isindependent of distance from the reference line, which may provide someadvantages, one limitation of the orthogonal projection is thatinformation relating to surfaces perpendicular to the road element maybe lost. This is illustrated by the side depth map of FIG. 16B obtainedusing the orthogonal projection.

If a non-orthogonal projection is used, e.g. at 45 degrees, then suchinformation relating to surfaces perpendicular to the road element maybe preserved. This is shown by FIGS. 16C and 16D. FIG. 16C illustrates a45 degree projection on to a reference plane defined as beingperpendicular to the road reference line once again. As FIG. 16D shows,the side depth map obtained using this projection includes moreinformation regarding those surfaces of the objects that areperpendicular to the road element. By using a non-orthogonal projection,information about such perpendicular surfaces may be captured by thedepth map data, but without needing to include additional data channels,or otherwise increase storage capacity. It will be appreciated thatwhere such a non-orthogonal projection is used for the depth map data ofthe localisation reference data, then a corresponding projection shouldbe used for real time sensed data to which it is to be compared.

Each pixel of the depth map data for the localisation reference data isbased upon a group of sensed measurement, e.g. laser measurements. Thesemeasurements correspond to the sensor measurements indicative of adistance of an object from the reference plane along the relevantpredetermined direction at the position of the pixel. Due to the way inwhich data is compressed, a group of sensor measurements will be mappedto a particular pixel. Rather than determine a depth value to beassociated with the pixel that corresponds to an average of thedifferent distances according to the group of sensor measurements, ithas been found that greater accuracy may be obtained where the closestdistance from among the distances corresponding to the various sensormeasurements is used for the pixel depth value. It is important that thedepth value of a pixel accurately reflects the distance from thereference plane to the closest surface of an object. This is of greatestinterest when determining the position of a vehicle accurately, in amanner that will minimise risk of collision. If an average of a group ofsensor measurements is used to provide the depth value for a pixel,there is a likelihood that the depth value will suggest a greaterdistance to an object surface than is in fact the case at the pixelposition. This is because one object may transiently be located betweenthe reference plane and another more distant object, e.g. a tree may belocated in front of a building. In this situation, some sensormeasurements used to provide a pixel depth value will relate to thebuilding, and others to the tree, as a result of the area over whichsensor measurements map to the pixel extending beyond the tree on a sideor sides thereof. The Applicant has recognised that it is safest andmost reliable to take the closest of the various sensor measurements asthe depth value associated with the pixel in order to ensure thatdistance to the surface of the closest object is reliably captured, inthis case the tree. Alternatively, a distribution of the sensormeasurements for the pixel may be derived, and a closest mode taken toprovide the pixel depth. This will provide a more reliable indication ofdepth for the pixel, in a similar manner to a closest distance.

As described above, the pixels of the depth map data for thelocalisation reference data include a depth channel, which includes dataindicative of a depth from the position of the pixel in the referenceplane to the surface of an object. One or more additional pixel channelsmay be included in the localisation reference data. This will result ina multi-channel or layer depth map, and hence raster image. In somepreferred embodiments a second channel includes data indicative of alaser reflectivity of the object at the position of the pixel, and athird channel includes data indicative of a radar reflectivity of theobject at the pixel position.

Each pixel has a position corresponding to a particular distance alongthe road reference line (x-direction), and a height above the roadreference line (y-direction). The depth value associated with the pixelin a first channel c1 is indicative of the distance of the pixel in thereference plane along a predetermined direction (which may be orthogonalor non-orthogonal to the reference plane depending upon the projectionused) to the surface of a closest object (preferably corresponding tothe closest distance of a group of sensed measurements used to obtainthe pixel depth value). Each pixel may, in a second channel c2, have alaser reflectivity value indicative of a mean local reflectivity oflaser points at around the distance c1 from the reference plane. In athird channel c3, the pixel may have a radar reflectivity valueindicative of a mean local reflectivity of radar points at around c1distance from the reference plane. This is shown, for example, in FIG.17. The multi-channel format allows a significantly greater amount ofdata to be included in the depth map. Further possible channels that maybe used are an object thickness, (which may be used to restoreinformation about surfaces perpendicular to the road trajectory where anorthogonal projection is used), reflected point density, and colourand/or texture (e.g. obtained from a camera used in providing thereference scan data).

Although the invention has been described in relation to embodiments inwhich the depth map of the localisation reference data relates to theenvironment to the lateral sides of a road, it has been realised thatthe use of a depth map of a different configuration may be useful toassist in positioning a vehicle at a cross-roads. These furtherembodiments may be used in conjunction with the side depth maps forregions away from the cross-roads.

In some further embodiments, a reference line is defined in the form ofa circle. In other words, the reference line is non-linear. The circleis defined by a given radius centred on a centre of a cross-roads of thedigital map. The radius of the circle may be selected depending upon theside of the cross-roads. The reference plane may be defined as a 2dimensional surface perpendicular to this reference line. A (circular)depth map may then be defined, in which each pixel includes a channelindicative of a distance from the position of the pixel in the referenceplane to the surface of an object i.e. a depth value, along apredetermined direction in the same manner as when a linear referenceline is used. The projection onto the reference plane may similarly beorthogonal, or non-orthogonal, and each pixel may have multiplechannels. The depth value of a given pixel is preferably based upon aclosest sensed distance to an object.

FIG. 18 indicates circular and linear reference lines which may be usedin constructing depth maps at a cross-roads, and away from thecross-roads respectively. FIG. 19A illustrates the way in which objectsmay be projected on to the circular depth map at different angularpositions. FIG. 19B indicates the projection of each of the objects onto the reference plane to provide the depth map, using an orthogonalprojection.

The way in which a depth map of the localisation reference data, whethercircular or otherwise, may be compared to real time sensor data obtainedfrom a vehicle in order to determine a longitudinal alignment offsetbetween the reference and real time sensed data has been described. Insome further embodiments a lateral alignment offset is also obtained.This involves a series of steps which may be performed in the imagedomain.

Referring to an example using side depth maps, in a first step of theprocess, a longitudinal alignment offset between the reference and realtime sensor data based side depth maps is determined, in the mannerpreviously described. The depth maps are shifted relative to one anotheruntil they are longitudinally aligned. Next the reference depth map i.e.raster image is cropped so as to correspond in size to the depth mapbased upon real time sensor data. The depth values of pixels in thecorresponding positions of the thus aligned reference and real timesensor based side depth maps i.e. the value of the depth channel of thepixels, is then compared. The difference in the depth values of eachcorresponding pair of pixels indicates the lateral offset of the pixels.This may be assessed by consideration of the colour difference of thepixels, where the depth value of each pixel is represented by a colour.The most common lateral offset thus determined between correspondingpairs of pixels (the mode difference), is determined, and taken tocorrespond to the lateral alignment offset of the two depth maps. Themost common lateral offset may be obtained using a histogram of thedepth differences between pixels. Once the lateral offset has beendetermined, it may be used to correct a deemed lateral position of thevehicle on the road.

FIG. 20A illustrates a reference depth map, i.e. image, and acorresponding depth map or image based on real time sensor data from avehicle that may be compared to determine a lateral offset alignment ofthe depth maps. As FIG. 20B illustrates, first the images are shiftedrelative to one another to longitudinally align them. Next, aftercropping of the reference image, a histogram of the difference in pixeldepth value for corresponding pixels in the two depth maps is used todetermine the lateral alignment offset between the depth maps—FIG. 20C.FIG. 20D illustrates how this may enable the longitudinal position, andthen the lateral position of the vehicle on the road to be corrected.

Once a lateral alignment offset between reference and real time databased depth maps has been obtained, the heading of a vehicle may also becorrected. It has been found that where there is an offset between theactual and deemed headings of a vehicle, this will results in anon-constant lateral alignment offset being determined betweencorresponding pixels in the reference and real time sensed data baseddepth maps as a function of longitudinal distance along the depth map.

FIG. 21A illustrates a set of vertical slices through correspondingportions of a reference depth map image (upper), and real time sensorbased depth map image (lower). The mean difference in pixel depth valuefor the corresponding pixels in each slice (i.e. the lateral alignmentoffset), is plotted (y axis) with respect to longitudinal distance alongthe map/image (x axis). Such a plot is shown in FIG. 21B. A functiondescribing the relationship between the mean pixel depth distance andlongitudinal distance along the depth map may then be derived bysuitable regression analysis. The gradient of this function isindicative of the heading offset of the vehicle.

The depth maps used in embodiments of the present invention may betransformed so as to always be relative to a straight reference line,i.e. so as to be linearly referenced images, e.g. as described in WO2009/045096 A1. This has an advantage as shown in FIG. 22. At the leftside of FIG. 22 is an image of a curved road. To mark the centreline ofthe curved road, a number of marks 1102 have to be placed. At the righthand side of FIG. 22, a corresponding linearly referenced image is showncorresponding to the curved road in the left side of the drawing. Toobtain the linearly referenced image, the centreline of the curved roadis mapped to the straight reference line of the linearly referencedimage. In view of this transformation, the reference line can now bedefined simply by two end points 1104 and 1106.

While on perfectly straight roads, the shift calculated from thecomparison of the reference and real-time depth maps can be directlyapplied, the same is not possible on curved roads due to the non-linearnature of the linearisation procedure used to produce the linearlyreferenced images. FIGS. 23A and 23B show a computationally efficientmethod for establishing the position of the vehicle in a non-linearenvironment through a series of incremental independent linear updatesteps. As is shown in FIG. 23A, the method involves applying alongitudinal correction, then a lateral correction and then a headingcorrection in a series of incremental, independent linear update steps.In particular, in step (1) a longitudinal offset is determined usingvehicle sensor data and a reference depth map that is based on thecurrent deemed position of the vehicle relative to a digital map (e.g.obtained using GPS). The longitudinal offset is then applied to adjustthe deemed position of the vehicle relative to a digital map and thereference depth map is recomputed based on the adjusted position. Then,in step (2), a lateral offset is determined using the vehicle sensordata and the recomputed reference depth map. The lateral offset is thenapplied to further adjust the deemed position of the vehicle relative tothe digital map and the reference depth map is again recomputed based onthe adjusted position. Finally, at step (3), a heading offset or skew isdetermined using the vehicle sensor data and the recomputed referencedepth map. The heading offset is then applied to further adjust thedeemed position of the vehicle relative to the digital map and thereference depth map is again recomputed based on the adjusted position.These steps are repeated as many times as is needed for there to besubstantially zero longitudinal, lateral and heading offset between thereal time depth map and the reference depth map. FIG. 23B shows thesequential and repeated application of a longitudinal, lateral andheading offsets to a point cloud generated from vehicle sensor datauntil that point cloud substantially aligns with a point cloud generatedfrom the reference depth map.

A series of exemplary use cases for localisation reference data are alsodepicted.

For example, rather than using a depth map of the localisation referencedata for the purposes of comparison to a depth map based on real timesensor data, in some embodiments, the depth map of the localisationreference data is used to generate a reference point cloud, including aset of data points in a three dimensional coordinate system, each pointrepresenting a surface of an object in the environment. This referencepoint cloud may be compared to a corresponding three dimensional pointcloud based upon real time sensor data obtained by vehicle sensors. Thecomparison may be used to determine an alignment offset between thedepth maps, and hence to adjust the determined position of the vehicle.

The reference depth map may be used to obtain a reference 3D point cloudthat may be compared to a corresponding point cloud based upon real-timesensor data of a vehicle, whatever type of sensors that vehicle has.While the reference data may be based upon sensor data obtained fromvarious types of sensor, including laser scanners, radar scanners, andcameras, a vehicle may not have a corresponding set of sensors. A 3Dreference point cloud may be constructed from the reference depth mapthat may be compared to a 3D point cloud obtained based on theparticular type of real time sensor data available for a vehicle.

For example, where the depth map of the reference localisation dataincludes a channel indicative of radar reflectivity, this may be takeninto account in generating a reference point cloud that may be comparedto 3D point cloud obtained using real time sensor data of a vehiclewhich has only a radar sensor. The radar reflectivity data associatedwith pixels helps to identify those data points which should be includedin the 3D reference point cloud, i.e. which represent surfaces ofobjects that the vehicle radar sensor would be expected to detect.

In another example, the vehicle may have only a camera or cameras forproviding real time sensor data. In this case, data from a laserreflectivity channel of the reference depth map may be used to constructa 3D reference point cloud including data points relating only tosurfaces that may be expected to be detected by the camera(s) of thevehicle under current conditions. For example, when it is dark, onlyrelatively reflective objects should be included.

A 3D point cloud based upon real time sensed data of a vehicle may beobtained as desired. Where the vehicle includes only a single camera asa sensor, a “structure from motion” technique may be used, in which asequence of images from the camera are used to reconstruct a 3D scene,from which a 3D point cloud may be obtained. Where the vehicle includesstereo cameras, a 3D scene may be generated directly, and used toprovide the 3-dimensional point cloud. This may be achieved using adisparity based 3D model.

In yet other embodiments, rather than comparing the reference pointcloud to the real time sensor data point cloud, the reference pointcloud is used to reconstruct an image that would be expected to be seenby a camera or cameras of the vehicle. The images may then be compared,and used to determine an alignment offset between the images, which inturn, may be used to correct a deemed position of the vehicle.

In these embodiments, additional channels of the reference depth map maybe used as described above to reconstruct an image based on includingonly those points in the 3-dimensional reference point cloud that wouldbe expected to be detected by the camera(s) of the vehicle. For example,in the dark, the laser reflectivity channel may be used to select thosepoints for inclusion in the 3-dimensional point cloud that correspond tothe surfaces of objects that could be detected by the camera(s) in thedark. It has been found that the use of a non-orthogonal projection onto the reference plane when determining the reference depth map isparticularly useful in this context, preserving more information aboutsurfaces of objects which may still be detectable in the dark.

FIG. 24 depicts an exemplary system in accordance with embodiments ofthe invention in which data collected by one or more vehicle sensors:laser; camera; and radar, is used to generate an “actual footprint” ofthe environment as seen by the vehicle. The “actual footprint” iscompared, i.e. correlated, to a corresponding “reference footprint” thatis determined from reference data associated with a digital map, whereinthe reference data includes at least a distance channel, and may includea laser reflectivity channel and/or a radar reflectivity channel, as isdiscussed above. Through this correlation, the position of the vehiclecan be accurately determined relative to the digital map.

In a first example use case, as depicted in FIG. 25A, an actualfootprint is determined from a laser-based range sensor, e.g. LIDARsensor, in the vehicle and correlated to a reference footprintdetermined from data in the distance channel of the reference data, soas to achieve continuous positioning of the vehicle. A first approach isshown in FIG. 25B in which the laser point cloud as determined by thelaser-based range sensor is converted into a depth map of the sameformat as the reference data, and the two depth map images are compared.A second, alternative approach is shown in FIG. 25C in which a laserpoint cloud is reconstructed from the reference data, and thisreconstructed point cloud compared to the laser point cloud as seen bythe vehicle.

In a second example use case, as depicted in FIG. 26A, an actualfootprint is determined from a camera in the vehicle and correlated to areference footprint determined from data in the distance channel of thereference data, so as to achieve continuous positioning of the vehicle,although only during the day. In other words, in this example use case areference depth map is used to construct a 3D point cloud or view thatis then compared to a 3D scene or view obtained from multiple vehiclecameras or a single vehicle camera. A first approach is shown in FIG.26B in which stereo vehicle cameras are used to build a disparity based3D model, which is then used to construct a 3D point cloud forcorrelation with the 3D point cloud constructed from the reference depthmap. A second approach is shown in FIG. 26C in which a sequence ofvehicle camera images is used to construct a 3D scene, which is thenused to construct a 3D point cloud for correlation with the 3D pointcloud constructed from the reference depth map. Finally, a thirdapproach is shown in FIG. 25D in which a vehicle camera image iscompared with a view created from the 3D point cloud constructed fromthe reference depth map.

In a third example use case, as depicted in FIG. 27A, is a modificationto the second example use case wherein laser reflectivity data of thereference data, which is in a channel of the depth map, can be used toconstruct a 3D point cloud or view that may be compared to a 3D pointcloud or view based on images captured by one or more cameras. A firstapproach is shown in FIG. 27B, wherein a sequence of vehicle cameraimages is used to construct a 3D scene, which is then used to constructa 3D point cloud for correlation with the 3D point cloud constructedfrom the reference depth map (using both the distance and laserreflectivity channels). A second approach is shown in FIG. 27C in whicha vehicle camera image is compared with a view created from the 3D pointcloud constructed from the reference depth map (again using both thedistance and laser reflectivity channels).

In a fourth example use case, as depicted in FIG. 28A, an actualfootprint is determined from a radar-based range sensor in the vehicleand correlated to a reference footprint determined from data in thedistance and radar reflectivity channels of the reference data, so as toachieve sparse positioning of the vehicle. A first approach is shown inFIG. 28B, wherein reference data is used to reconstruct a 3D scene anddata in the radar reflectivity channel is used to leave only theradar-reflective points. This 3D scene is then correlated with the radarpoint cloud as seen by the car.

It will of course be understood that the various use cases could be usedtogether, i.e. fused, to allow for a more precise localisation of thevehicle relative to the digital map.

The method of correlating vehicle sensor data to reference data in orderto determine the position of the vehicle, e.g. as discussed above, willnow be described with reference to FIGS. 29 to 32B. FIG. 29 depicts thevarious coordinate systems that are used in method: the Local coordinatesystem (Local CS); the Car Frame coordinate system (CF CS); and theLinearly Referenced coordinate system (LR CS) along the trajectory ofthe car. Another coordinate system, although not depicted, is the WorldGeodetic System (WGS) in which positions are given as latitude,longitude coordinate pairs as known in the art. The general method isshown in FIG. 30, with details of the steps performed to determine thelaser point cloud being shown in FIG. 31. FIG. 32A shows a firstexemplary method to perform the correlation step of FIG. 30 in which theposition of the vehicle is corrected by image correlation, e.g. betweena depth map raster image of the reference data and a corresponding depthmap raster image created from the vehicle sensor data. FIG. 32B shows asecond exemplary method to perform the correlation step of FIG. 30 inwhich the position is of the vehicle is corrected by 3D correlation,e.g. between a 3D scene constructed from the reference data and the 3Dscene captured by the vehicle sensors.

Any of the methods in accordance with the present invention may beimplemented at least partially using software e.g. computer programs.The present invention thus also extends to a computer program comprisingcomputer readable instructions executable to perform, or to cause anavigation device to perform, a method according to any of the aspectsor embodiments of the invention. Thus, the invention encompasses acomputer program product that, when executed by one or more processors,cause the one or more processors to generate suitable images (or othergraphical information) for display on a display screen. The inventioncorrespondingly extends to a computer software carrier comprising suchsoftware which, when used to operate a system or apparatus comprisingdata processing means causes, in conjunction with said data processingmeans, said apparatus or system to carry out the steps of the methods ofthe present invention. Such a computer software carrier could be anon-transitory physical storage medium such as a ROM chip, CD ROM ordisk, or could be a signal such as an electronic signal over wires, anoptical signal or a radio signal such as to a satellite or the like. Thepresent invention provides a machine readable medium containinginstructions which when read by a machine cause the machine to operateaccording to the method of any of the aspects or embodiments of theinvention.

Where not explicitly stated, it will be appreciated that the inventionin any of its aspects may include any or all of the features describedin respect of other aspects or embodiments of the invention to theextent they are not mutually exclusive. In particular, while variousembodiments of operations have been described which may be performed inthe method and by the apparatus, it will be appreciated that any one ormore or all of these operations may be performed in the method and bythe apparatus, in any combination, as desired, and as appropriate.

1. A method of determining a position of a vehicle relative to a digitalmap, the digital map comprising data representative of navigableelements of a navigable network along which the vehicle is travelling,the method comprising: obtaining localisation reference data associatedwith the digital map for a deemed current position of the vehicle alonga navigable element of the navigable network, wherein the locationreference data comprises at least one depth map indicative of anenvironment around the vehicle projected on to a reference plane, thereference plane being defined by a reference line associated with thenavigable element, each pixel of the at least one depth map beingassociated with a position in the reference plane associated with thenavigable element along which the vehicle is travelling, and the pixelincluding a depth channel representing the distance to a surface of anobject in the environment along a predetermined direction from theassociated position of the pixel in the reference plane; using thelocalisation reference data to determine a reference point cloudindicative of the environment around the vehicle, the reference pointcloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment; determining real time scan data byscanning the environment around the vehicle using at least one sensor,the real time scan data comprising a point cloud indicative of theenvironment around the vehicle, the point cloud including a set ofsecond data points in the three-dimensional coordinate system, whereineach data point represents a surface of an object in the environment asdetermined using the at least one sensor; calculating a correlationbetween the point cloud of the real time scan data and the point cloudof the obtained localisation reference data to determine an alignmentoffset between the point clouds; and using the determined alignmentoffset to adjust the deemed current position to determine the positionof the vehicle relative to the digital map.
 2. The method as claimed inclaim 1, wherein the point cloud of the real time scan data is obtainedusing one or more sensors associated with the vehicle.
 3. The method asclaimed in claim 2, wherein the one or more sensors comprise a set ofone or more laser scanners, radar scanners and/or cameras.
 4. The methodas claimed in claim 1, wherein images obtained from one or more camerasassociated with the vehicle are used to construct a three dimensionalscene indicative of the environment around the vehicle, and the pointcloud of the real time scan data is obtained using the three dimensionalscene.
 5. The method as claimed in claim 1, wherein the point cloud ofthe real time scan data is determined by obtaining a sequence of twodimensional images from a camera associated with the vehicle as thevehicle travels along the navigable element or through the junction,using the sequence of two dimensional images to construct a threedimensional scene indicative of the environment around the vehicle, andusing the three dimensional scene to obtain the point cloud of the realtime scan data.
 6. The method as claimed in claim 1, wherein the pointcloud of the real time scan data is determined by obtaining images fromstereo cameras associated with the vehicle, using the images toconstruct a three dimensional scene indicative of the environment aroundthe vehicle, and using the three dimensional scene to obtain the pointcloud of the real time scan data.
 7. The method as claimed in claim 1,comprising including those points in the reference point cloud thatwould be expected to be detected by one or more sensors of the typeassociated with the vehicle and/or expected to be detected under currentconditions.
 8. The method as claimed in claim 7, wherein including thosepoints in the reference point cloud that would be expected to bedetected by the one or more sensors of the type associated with thevehicle and/or expected to be detected under current conditionscomprises using data of at least one reflectivity data channelassociated with the pixels of the depth map of the localisationreference data.
 9. The method as claimed in claim 8, wherein the atleast one reflectivity data channel comprises a laser reflectivity datachannel and/or a radar reflectivity data channel.
 10. A method ofdetermining a position of a vehicle relative to a digital map, thedigital map comprising data representative of navigable elements of anavigable network along which the vehicle is travelling, the methodcomprising: obtaining localisation reference data associated with thedigital map for a deemed current position of the vehicle along anavigable element of the navigable network, wherein the locationreference data comprises at least one depth map indicative of anenvironment around the vehicle projected on to a reference plane, thereference plane being defined by a reference line associated with thenavigable element, each pixel of the at least one depth map beingassociated with a position in the reference plane associated with thenavigable element along which the vehicle is travelling, and the pixelincluding a depth channel representing the distance to a surface of anobject in the environment along a predetermined direction from theassociated position of the pixel in the reference plane; using thelocalisation reference data to determine a reference point cloudindicative of the environment around the vehicle, the reference pointcloud including a set of first data points in a three-dimensionalcoordinate system, wherein each first data point represents a surface ofan object in the environment; using the reference point cloud toreconstruct a reference view that would be expected to be obtained byone or more cameras associated with the vehicle when traversing thenavigable element under applicable conditions; determining a real timeview of the environment around the vehicle using the one or morecameras; calculating a correlation between the reference view and thereal time view obtained by the one or more cameras to determine analignment offset between the views; and using the determined alignmentoffset to adjust the deemed current position of the vehicle to determinethe position of the vehicle relative to the digital map.
 11. The methodas claimed in claim 10, comprising including those points in thereference point cloud that would be expected to be detected by the oneor more camera and/or expected to be detected under current conditions.12. The method as claimed in claim 11, wherein including those points inthe reference point cloud that would be expected to be detected by theone or more camera and/or expected to be detected under currentconditions comprises using data of at least one reflectivity datachannel associated with the pixels of the depth map of the localisationreference data.
 13. The method as claimed in claim 12, wherein the atleast one reflectivity data channel comprises a laser reflectivity datachannel and/or a radar reflectivity data channel.
 14. A non-transitorycomputer readable medium comprising computer readable instructionsexecutable to cause a system to perform a method as claimed in claim 1.15-16. (canceled)